2024-06-03T05:04:55.068 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-03T05:04:55.068 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436 --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/valgrind} --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/orig.config.yaml 2024-06-03T05:04:55.108 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-03T05:04:55.195 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436 branch: squid description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/valgrind} email: ceph-qa@ceph.com first_in_suite: false job_id: '7738436' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(OSDMAP_FLAGS\) - \(SMALLER_PGP_NUM\) - \(POOL_APP_NOT_ENABLED\) - \(SLOW OPS\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 5132 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 20949/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: dfcc4b3c2c55532c4c04fa47551c9df1dffdf746 targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIwl5p2Yl/+R2k5UW83Q/3ZbmRKpKyh0MBcBIImYmNkv+QajZidIFJySEPOBL0apnqYE/VG5YuLn3bkuVHJKkvQ= smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAFjBxwOlDeyKlvgiJAe0zBy0004ZNp0GEi6F0ppdc4Vwd3os3jdqmrzsToXFZqcM8uZMVOhA9EbQDe7+4l3Gos= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - mon_recovery: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-02_21:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-03T05:04:55.195 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa; will attempt to use it 2024-06-03T05:04:55.196 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks 2024-06-03T05:04:55.196 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-03T05:04:55.198 INFO:teuthology.task.internal:Checking packages... 2024-06-03T05:04:55.218 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-03T05:04:55.218 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-03T05:04:55.219 INFO:teuthology.packaging:ref: None 2024-06-03T05:04:55.219 INFO:teuthology.packaging:tag: None 2024-06-03T05:04:55.219 INFO:teuthology.packaging:branch: squid 2024-06-03T05:04:55.219 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T05:04:55.219 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-03T05:04:55.535 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2952.gdfcc4b3c 2024-06-03T05:04:55.537 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-03T05:04:55.550 INFO:teuthology.task.internal:no buildpackages task found 2024-06-03T05:04:55.550 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-03T05:04:55.575 INFO:teuthology.task.internal:Saving configuration 2024-06-03T05:04:55.590 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-03T05:04:55.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-03T05:04:55.624 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-03 04:59:34.711572', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIwl5p2Yl/+R2k5UW83Q/3ZbmRKpKyh0MBcBIImYmNkv+QajZidIFJySEPOBL0apnqYE/VG5YuLn3bkuVHJKkvQ='} 2024-06-03T05:04:55.647 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-03 04:59:34.709615', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAFjBxwOlDeyKlvgiJAe0zBy0004ZNp0GEi6F0ppdc4Vwd3os3jdqmrzsToXFZqcM8uZMVOhA9EbQDe7+4l3Gos='} 2024-06-03T05:04:55.647 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-03T05:04:55.655 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-06-03T05:04:55.655 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-06-03T05:04:55.655 INFO:teuthology.run_tasks:Running task console_log... 2024-06-03T05:04:55.720 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f33c75a7ac0>, signals=[15]) 2024-06-03T05:04:55.720 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-03T05:04:55.728 INFO:teuthology.task.internal:Opening connections... 2024-06-03T05:04:55.729 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:04:55.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:04:55.808 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:04:55.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:04:55.884 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-03T05:04:55.892 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-06-03T05:04:55.911 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-06-03T05:04:55.911 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-06-03T05:04:55.969 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-06-03T05:04:55.969 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-03T05:04:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-03T05:04:55.971 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-06-03T05:04:55.990 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-06-03T05:04:56.009 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-06-03T05:04:56.009 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-06-03T05:04:56.067 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-06-03T05:04:56.067 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-06-03T05:04:56.067 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-06-03T05:04:56.067 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-06-03T05:04:56.067 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-06-03T05:04:56.067 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-06-03T05:04:56.068 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-03T05:04:56.068 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-06-03T05:04:56.068 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-06-03T05:04:56.068 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-03T05:04:56.068 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-06-03T05:04:56.068 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-03T05:04:56.068 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-03T05:04:56.068 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-03T05:04:56.068 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-06-03T05:04:56.090 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-03T05:04:56.101 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-03T05:04:56.109 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-03T05:04:56.110 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-06-03T05:04:56.112 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-06-03T05:04:56.129 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-03T05:04:56.137 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-03T05:04:56.137 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-06-03T05:04:56.170 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-06-03T05:04:56.195 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-03T05:04:56.252 INFO:teuthology.run_tasks:Running task kernel... 2024-06-03T05:04:56.268 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-03T05:04:56.268 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-03T05:04:56.269 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-03T05:04:56.269 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-03T05:04:56.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:04:56.286 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-06-03T05:04:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:04:56.287 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-06-03T05:04:56.344 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-437.el9.x86_64 2024-06-03T05:04:56.344 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-437.el9.x86_64 2024-06-03T05:04:56.344 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-06-03T05:04:56.345 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-437.el9.x86_64 2024-06-03T05:04:56.345 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-437.el9.x86_64 2024-06-03T05:04:56.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-06-03T05:04:58.655 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.1 MB 00:00 2024-06-03T05:04:58.695 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.1 MB 00:00 2024-06-03T05:05:00.431 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-06-03T05:05:00.588 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-06-03T05:05:04.798 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.2 MB 00:00 2024-06-03T05:05:04.988 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-06-03T05:05:06.324 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 17 kB 00:00 2024-06-03T05:05:06.372 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 102 kB/s | 17 kB 00:00 2024-06-03T05:05:07.682 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-06-03T05:05:08.028 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-06-03T05:05:13.207 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 9.8 kB/s | 1.7 kB 00:00 2024-06-03T05:05:13.400 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-06-03T05:05:15.296 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-03T05:05:15.296 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-03T05:05:15.437 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:05:15.443 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:05:15.444 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:05:15.445 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-06-03T05:05:15.445 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:05:15.445 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 80 M 2024-06-03T05:05:15.445 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 124 M 2024-06-03T05:05:15.445 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:05:15.490 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-03T05:05:15.490 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-03T05:05:15.582 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:05:15.594 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:05:15.594 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-06-03T05:05:15.594 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:05:15.594 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:05:15.594 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-03T05:05:15.594 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:05:15.595 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-03T05:05:15.595 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-03T05:05:15.595 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-03T05:05:15.595 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:05:15.595 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:05:15.595 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:05:15.595 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-06-03T05:05:15.595 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:05:15.596 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 80 M 2024-06-03T05:05:15.596 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 124 M 2024-06-03T05:05:15.596 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:05:15.919 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 5.1 MB/s | 893 kB 00:00 2024-06-03T05:05:16.024 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 2.9 MB/s | 893 kB 00:00 2024-06-03T05:05:16.161 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-06-03T05:05:16.611 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 40 MB/s | 35 MB 00:00 2024-06-03T05:05:16.878 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): kernel-modules-core-5.14.0-452.el9.x86_6 30 MB/s | 29 MB 00:00 2024-06-03T05:05:16.878 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:05:16.878 INFO:teuthology.orchestra.run.smithi038.stdout:Total 63 MB/s | 80 MB 00:01 2024-06-03T05:05:17.183 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-06-03T05:05:17.607 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:05:17.742 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 17 MB/s | 29 MB 00:01 2024-06-03T05:05:18.012 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:05:18.012 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:05:18.517 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 12 MB/s | 35 MB 00:02 2024-06-03T05:05:18.517 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:05:18.517 INFO:teuthology.orchestra.run.smithi064.stdout:Total 26 MB/s | 80 MB 00:03 2024-06-03T05:05:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:05:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:05:19.242 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:05:19.676 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:05:19.676 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:05:20.641 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:05:20.641 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:05:21.540 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:05:22.226 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-03T05:05:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T05:05:23.203 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:05:23.873 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-03T05:05:23.967 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T05:05:25.119 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T05:05:25.265 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T05:05:26.772 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T05:05:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T05:05:30.364 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T05:05:30.412 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:05:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T05:05:32.282 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:05:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:05:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:06:06.704 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:06:06.745 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:06:07.076 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:06:07.076 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-03T05:06:07.077 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T05:06:07.077 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T05:06:07.931 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:06:07.931 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:06:07.931 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:06:07.932 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-03T05:06:07.932 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-03T05:06:07.932 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-03T05:06:07.932 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-03T05:06:07.932 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:06:07.932 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:06:08.336 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T05:06:08.417 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T05:06:08.418 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T05:06:08.418 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-03T05:06:08.418 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-03T05:06:08.419 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-03T05:06:08.419 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-03T05:06:08.419 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-06-03T05:06:08.419 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T05:06:08.472 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T05:06:08.473 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T05:06:08.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-06-03T05:06:08.504 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-06-03T05:06:08.504 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-06-03T05:06:08.504 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.06 2024-06-03T05:06:08.504 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 68.el9 2024-06-03T05:06:08.504 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-06-03T05:06:08.504 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-03T05:06:08.504 INFO:teuthology.orchestra.run.smithi038.stdout:Group : Unspecified 2024-06-03T05:06:08.504 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 8264254 2024-06-03T05:06:08.504 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : builder@centos.org 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-03T05:06:08.505 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-06-03T05:06:08.506 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:06:08.506 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-06-03T05:06:08.506 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-03T05:06:08.507 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 5.14.0-452.el9.x86_64 2024-06-03T05:06:08.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-03T05:06:08.728 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:06:08.770 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:06:09.210 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-06-03T05:06:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:06:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-03T05:06:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T05:06:09.273 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T05:06:10.052 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T05:06:10.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:06:10.053 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:06:10.053 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-03T05:06:10.053 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-03T05:06:10.053 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-03T05:06:10.053 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-03T05:06:10.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:06:10.053 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:06:10.381 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T05:06:10.464 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T05:06:10.464 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T05:06:10.465 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-03T05:06:10.465 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-03T05:06:10.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-03T05:06:10.465 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-06-03T05:06:10.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-06-03T05:06:10.465 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T05:06:10.518 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T05:06:10.518 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T05:06:10.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-06-03T05:06:10.555 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-06-03T05:06:10.555 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-06-03T05:06:10.555 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-06-03T05:06:10.555 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-06-03T05:06:10.555 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-06-03T05:06:10.555 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-03T05:06:10.555 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-06-03T05:06:10.555 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-03T05:06:10.556 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-03T05:06:10.557 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-06-03T05:06:10.557 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:06:10.557 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-06-03T05:06:10.557 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-03T05:06:10.558 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-452.el9.x86_64 2024-06-03T05:06:10.558 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-03T05:06:10.698 INFO:teuthology.orchestra.run.smithi038.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-03T05:06:10.724 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-06-03T05:06:10.727 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-06-03T05:06:10.742 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.3Pa2hoLhnt 2024-06-03T05:06:10.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3Pa2hoLhnt 2024-06-03T05:06:10.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.3Pa2hoLhnt 2024-06-03T05:06:10.946 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.3Pa2hoLhnt is 6KB 2024-06-03T05:06:10.994 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.3Pa2hoLhnt 2024-06-03T05:06:11.010 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2024-06-03T05:06:11.072 INFO:teuthology.orchestra.run.smithi038.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-03T05:06:11.072 INFO:teuthology.orchestra.run.smithi038.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-03T05:06:11.072 INFO:teuthology.orchestra.run.smithi038.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-03T05:06:11.072 INFO:teuthology.orchestra.run.smithi038.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-03T05:06:11.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-03T05:06:11.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-06-03T05:06:11.240 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-06-03T05:06:12.667 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-03T05:06:12.694 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-06-03T05:06:12.696 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-06-03T05:06:12.754 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.vCmUdqtwS0 2024-06-03T05:06:12.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vCmUdqtwS0 2024-06-03T05:06:12.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.vCmUdqtwS0 2024-06-03T05:06:12.973 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.vCmUdqtwS0 is 6KB 2024-06-03T05:06:13.022 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.vCmUdqtwS0 2024-06-03T05:06:13.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-06-03T05:06:13.098 INFO:teuthology.orchestra.run.smithi064.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-03T05:06:13.098 INFO:teuthology.orchestra.run.smithi064.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-03T05:06:13.099 INFO:teuthology.orchestra.run.smithi064.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-03T05:06:13.099 INFO:teuthology.orchestra.run.smithi064.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-03T05:06:13.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-03T05:06:13.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-06-03T05:06:41.254 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-03T05:06:41.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-06-03T05:06:41.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:06:43.261 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-06-03T05:06:43.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-06-03T05:06:43.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:07:17.919 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-06-03T05:07:26.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-06-03T05:07:26.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:07:41.262 DEBUG:teuthology.orchestra.remote:timed out 2024-06-03T05:07:45.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-06-03T05:07:50.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-06-03T05:07:50.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:07:50.524 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-06-03T05:07:51.119 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-06-03T05:07:51.119 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-452.el9.x86_64"... 2024-06-03T05:07:51.120 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-06-03T05:07:51.137 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-452.el9.x86_64 2024-06-03T05:07:51.138 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-03T05:07:51.138 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-03T05:07:51.138 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-03T05:07:52.138 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-03T05:07:52.138 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-03T05:07:52.202 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-06-03T05:07:52.245 DEBUG:teuthology.parallel:result is None 2024-06-03T05:07:57.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-06-03T05:07:57.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:07:58.478 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-06-03T05:07:58.937 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-06-03T05:07:58.937 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-452.el9.x86_64"... 2024-06-03T05:07:58.938 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-06-03T05:07:58.953 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-452.el9.x86_64 2024-06-03T05:07:58.953 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-03T05:07:58.953 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-03T05:07:58.954 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-03T05:07:59.955 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-06-03T05:07:59.955 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-03T05:08:00.104 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-06-03T05:08:00.136 DEBUG:teuthology.parallel:result is None 2024-06-03T05:08:00.136 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-03T05:08:00.146 INFO:teuthology.task.internal:Creating test directory... 2024-06-03T05:08:00.146 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-03T05:08:00.150 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-03T05:08:00.166 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-03T05:08:00.174 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-03T05:08:00.184 INFO:teuthology.task.internal:Creating archive directory... 2024-06-03T05:08:00.184 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-03T05:08:00.207 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-03T05:08:00.237 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-03T05:08:00.243 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-03T05:08:00.244 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-03T05:08:00.268 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-03T05:08:00.296 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:08:00.306 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:08:00.339 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:08:00.350 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T05:08:00.354 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-03T05:08:00.361 INFO:teuthology.task.internal:Configuring sudo... 2024-06-03T05:08:00.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-03T05:08:00.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-03T05:08:00.388 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-03T05:08:00.397 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-03T05:08:00.398 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-03T05:08:00.430 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-03T05:08:00.446 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T05:08:00.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T05:08:00.582 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T05:08:00.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T05:08:00.706 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:08:00.706 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-03T05:08:00.772 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T05:08:00.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T05:08:00.864 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T05:08:00.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T05:08:00.988 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:08:00.989 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-03T05:08:01.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-06-03T05:08:01.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-06-03T05:08:01.108 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T05:08:01.142 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T05:08:01.614 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-03T05:08:01.622 INFO:teuthology.task.internal:Starting timer... 2024-06-03T05:08:01.622 INFO:teuthology.run_tasks:Running task pcp... 2024-06-03T05:08:01.634 INFO:teuthology.run_tasks:Running task selinux... 2024-06-03T05:08:01.647 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-06-03T05:08:01.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-06-03T05:08:01.740 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-06-03T05:08:01.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-06-03T05:08:01.835 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-06-03T05:08:01.837 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-03T05:08:01.837 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-06-03T05:08:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-06-03T05:08:01.856 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-06-03T05:08:01.880 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-06-03T05:08:01.881 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi064.front.sepia.ceph.com': 'permissive'} 2024-06-03T05:08:01.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-03T05:08:01.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:08:01.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-03T05:08:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:08:01.952 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-03T05:08:01.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-06-03T05:08:01.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-06-03T05:08:02.025 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-03T05:08:02.036 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-03T05:08:02.038 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-03T05:08:02.158 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-03T05:08:02.169 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-03T05:08:02.170 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi064.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-03T05:12:27.916 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi064.front.sepia.ceph.com')] 2024-06-03T05:12:27.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-06-03T05:12:27.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:12:27.997 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-06-03T05:12:28.075 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-06-03T05:12:28.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-06-03T05:12:28.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T05:12:28.149 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-06-03T05:12:28.230 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-06-03T05:12:28.231 INFO:teuthology.run_tasks:Running task clock... 2024-06-03T05:12:28.243 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-03T05:12:28.243 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-03T05:12:28.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T05:12:28.246 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-03T05:12:28.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T05:12:28.276 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-03T05:12:28.291 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-03T05:12:28.302 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-03T05:12:28.317 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-06-03T05:12:28.318 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-03T05:12:28.328 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-06-03T05:12:28.343 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-03T05:12:28.344 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-06-03T05:12:28.356 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-06-03T05:12:28.358 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-03T05:12:28.370 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-03T05:12:28.385 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-03T05:12:28.419 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-06-03T05:12:28.423 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T05:12:28.423 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-06-03T05:12:28.423 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:12:28.423 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:12:28.423 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:12:28.423 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:12:28.442 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-06-03T05:12:28.444 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T05:12:28.444 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-06-03T05:12:28.444 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:12:28.444 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:12:28.444 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:12:28.444 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:12:28.445 INFO:teuthology.run_tasks:Running task install... 2024-06-03T05:12:28.454 DEBUG:teuthology.task.install:project ceph 2024-06-03T05:12:28.454 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-03T05:12:28.454 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-03T05:12:28.454 INFO:teuthology.task.install:Using flavor: default 2024-06-03T05:12:28.459 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-06-03T05:12:28.460 INFO:teuthology.task.install:extra packages: [] 2024-06-03T05:12:28.460 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-03T05:12:28.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T05:12:28.463 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-03T05:12:28.463 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T05:12:28.610 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-03T05:12:28.610 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-03T05:12:28.614 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-03T05:12:28.615 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-03T05:12:28.718 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-03T05:12:28.719 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:12:28.719 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-03T05:12:28.727 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-03T05:12:28.728 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:12:28.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-03T05:12:28.746 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-06-03T05:12:28.746 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-03T05:12:28.755 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-06-03T05:12:28.755 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-03T05:12:28.814 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-03T05:12:28.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-03T05:12:28.891 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-03T05:12:28.899 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-03T05:12:28.957 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-06-03T05:12:28.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-06-03T05:12:28.964 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-06-03T05:12:28.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-06-03T05:12:29.195 INFO:teuthology.orchestra.run.smithi038.stdout:45 files removed 2024-06-03T05:12:29.199 INFO:teuthology.orchestra.run.smithi064.stdout:45 files removed 2024-06-03T05:12:29.222 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2024-06-03T05:12:29.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-06-03T05:12:29.924 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-06-03T05:12:29.982 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-06-03T05:12:30.358 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-06-03T05:12:30.416 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-06-03T05:12:30.724 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-06-03T05:12:30.783 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-06-03T05:12:31.166 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.1 MB 00:00 2024-06-03T05:12:31.658 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.1 MB 00:00 2024-06-03T05:12:32.851 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 35 MB/s | 19 MB 00:00 2024-06-03T05:12:34.101 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-06-03T05:12:37.226 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.2 MB 00:00 2024-06-03T05:12:38.419 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-06-03T05:12:38.786 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 17 kB 00:00 2024-06-03T05:12:40.037 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 17 kB 00:00 2024-06-03T05:12:40.137 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-06-03T05:12:41.047 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-06-03T05:12:45.521 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-06-03T05:12:46.447 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-06-03T05:12:48.024 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:12:48.033 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:12:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:12:48.033 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:12:48.033 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:12:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-03T05:12:48.033 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-06-03T05:12:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-03T05:12:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-03T05:12:48.035 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout:Install 30 Packages 2024-06-03T05:12:48.036 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-06-03T05:12:48.037 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:12:48.037 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 60 M 2024-06-03T05:12:48.037 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:12:48.950 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:12:48.959 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:12:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:12:48.959 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:12:48.959 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:12:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-03T05:12:48.960 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-03T05:12:48.961 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout:Install 30 Packages 2024-06-03T05:12:48.962 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-06-03T05:12:48.963 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:12:48.963 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 60 M 2024-06-03T05:12:48.963 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:12:49.305 INFO:teuthology.orchestra.run.smithi064.stdout:(1/32): ceph-base-19.0.0-2946.g3d89accf.el9.x86 6.6 MB/s | 5.4 MB 00:00 2024-06-03T05:12:49.364 INFO:teuthology.orchestra.run.smithi064.stdout:(2/32): ceph-selinux-19.0.0-2946.g3d89accf.el9. 431 kB/s | 25 kB 00:00 2024-06-03T05:12:49.497 INFO:teuthology.orchestra.run.smithi064.stdout:(3/32): libcephfs2-19.0.0-2946.g3d89accf.el9.x8 5.5 MB/s | 747 kB 00:00 2024-06-03T05:12:49.589 INFO:teuthology.orchestra.run.smithi064.stdout:(4/32): libradosstriper1-19.0.0-2946.g3d89accf. 5.3 MB/s | 501 kB 00:00 2024-06-03T05:12:50.239 INFO:teuthology.orchestra.run.smithi064.stdout:(5/32): librgw2-19.0.0-2946.g3d89accf.el9.x86_6 8.2 MB/s | 5.3 MB 00:00 2024-06-03T05:12:50.431 INFO:teuthology.orchestra.run.smithi064.stdout:(6/32): ceph-common-19.0.0-2946.g3d89accf.el9.x 11 MB/s | 21 MB 00:01 2024-06-03T05:12:50.448 INFO:teuthology.orchestra.run.smithi064.stdout:(7/32): python3-ceph-argparse-19.0.0-2946.g3d89 216 kB/s | 45 kB 00:00 2024-06-03T05:12:50.532 INFO:teuthology.orchestra.run.smithi064.stdout:(8/32): python3-ceph-common-19.0.0-2946.g3d89ac 1.3 MB/s | 135 kB 00:00 2024-06-03T05:12:50.640 INFO:teuthology.orchestra.run.smithi064.stdout:(9/32): python3-rados-19.0.0-2946.g3d89accf.el9 2.9 MB/s | 321 kB 00:00 2024-06-03T05:12:50.741 INFO:teuthology.orchestra.run.smithi064.stdout:(10/32): python3-rbd-19.0.0-2946.g3d89accf.el9. 2.9 MB/s | 300 kB 00:00 2024-06-03T05:12:50.833 INFO:teuthology.orchestra.run.smithi064.stdout:(11/32): python3-rgw-19.0.0-2946.g3d89accf.el9. 1.1 MB/s | 99 kB 00:00 2024-06-03T05:12:50.898 INFO:teuthology.orchestra.run.smithi038.stdout:(1/32): ceph-base-19.0.0-2946.g3d89accf.el9.x86 4.0 MB/s | 5.4 MB 00:01 2024-06-03T05:12:50.982 INFO:teuthology.orchestra.run.smithi038.stdout:(2/32): ceph-selinux-19.0.0-2946.g3d89accf.el9. 302 kB/s | 25 kB 00:00 2024-06-03T05:12:50.983 INFO:teuthology.orchestra.run.smithi064.stdout:(12/32): ceph-radosgw-19.0.0-2946.g3d89accf.el9 4.1 MB/s | 10 MB 00:02 2024-06-03T05:12:51.025 INFO:teuthology.orchestra.run.smithi064.stdout:(13/32): python3-cephfs-19.0.0-2946.g3d89accf.e 281 kB/s | 162 kB 00:00 2024-06-03T05:12:51.050 INFO:teuthology.orchestra.run.smithi064.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 492 kB/s | 107 kB 00:00 2024-06-03T05:12:51.067 INFO:teuthology.orchestra.run.smithi064.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-06-03T05:12:51.117 INFO:teuthology.orchestra.run.smithi064.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-06-03T05:12:51.123 INFO:teuthology.orchestra.run.smithi038.stdout:(3/32): libcephfs2-19.0.0-2946.g3d89accf.el9.x8 5.2 MB/s | 747 kB 00:00 2024-06-03T05:12:51.142 INFO:teuthology.orchestra.run.smithi064.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-06-03T05:12:51.159 INFO:teuthology.orchestra.run.smithi064.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-06-03T05:12:51.185 INFO:teuthology.orchestra.run.smithi064.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 221 kB 00:00 2024-06-03T05:12:51.202 INFO:teuthology.orchestra.run.smithi064.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 857 kB/s | 36 kB 00:00 2024-06-03T05:12:51.216 INFO:teuthology.orchestra.run.smithi038.stdout:(4/32): libradosstriper1-19.0.0-2946.g3d89accf. 5.3 MB/s | 501 kB 00:00 2024-06-03T05:12:51.219 INFO:teuthology.orchestra.run.smithi064.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-06-03T05:12:51.253 INFO:teuthology.orchestra.run.smithi064.stdout:(22/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-06-03T05:12:51.570 INFO:teuthology.orchestra.run.smithi064.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 82 kB/s | 26 kB 00:00 2024-06-03T05:12:51.645 INFO:teuthology.orchestra.run.smithi064.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 697 kB/s | 309 kB 00:00 2024-06-03T05:12:51.695 INFO:teuthology.orchestra.run.smithi064.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-06-03T05:12:51.720 INFO:teuthology.orchestra.run.smithi064.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-06-03T05:12:51.821 INFO:teuthology.orchestra.run.smithi064.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-06-03T05:12:51.916 INFO:teuthology.orchestra.run.smithi038.stdout:(5/32): librgw2-19.0.0-2946.g3d89accf.el9.x86_6 7.6 MB/s | 5.3 MB 00:00 2024-06-03T05:12:51.929 INFO:teuthology.orchestra.run.smithi064.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-06-03T05:12:51.980 INFO:teuthology.orchestra.run.smithi064.stdout:(29/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.9 MB/s | 839 kB 00:00 2024-06-03T05:12:52.030 INFO:teuthology.orchestra.run.smithi064.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-06-03T05:12:52.116 INFO:teuthology.orchestra.run.smithi038.stdout:(6/32): ceph-common-19.0.0-2946.g3d89accf.el9.x 8.2 MB/s | 21 MB 00:02 2024-06-03T05:12:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:(7/32): python3-ceph-argparse-19.0.0-2946.g3d89 200 kB/s | 45 kB 00:00 2024-06-03T05:12:52.259 INFO:teuthology.orchestra.run.smithi038.stdout:(8/32): ceph-radosgw-19.0.0-2946.g3d89accf.el9. 3.8 MB/s | 10 MB 00:02 2024-06-03T05:12:52.284 INFO:teuthology.orchestra.run.smithi038.stdout:(9/32): python3-ceph-common-19.0.0-2946.g3d89ac 808 kB/s | 135 kB 00:00 2024-06-03T05:12:52.309 INFO:teuthology.orchestra.run.smithi038.stdout:(10/32): python3-cephfs-19.0.0-2946.g3d89accf.e 967 kB/s | 162 kB 00:00 2024-06-03T05:12:52.322 INFO:teuthology.orchestra.run.smithi064.stdout:(31/32): librados2-19.0.0-2946.g3d89accf.el9.x8 8.6 MB/s | 3.4 MB 00:00 2024-06-03T05:12:52.334 INFO:teuthology.orchestra.run.smithi038.stdout:(11/32): python3-rados-19.0.0-2946.g3d89accf.el 4.2 MB/s | 321 kB 00:00 2024-06-03T05:12:52.360 INFO:teuthology.orchestra.run.smithi038.stdout:(12/32): python3-rbd-19.0.0-2946.g3d89accf.el9. 3.9 MB/s | 300 kB 00:00 2024-06-03T05:12:52.372 INFO:teuthology.orchestra.run.smithi064.stdout:(32/32): librbd1-19.0.0-2946.g3d89accf.el9.x86_ 8.0 MB/s | 3.1 MB 00:00 2024-06-03T05:12:52.378 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:12:52.378 INFO:teuthology.orchestra.run.smithi064.stdout:Total 14 MB/s | 60 MB 00:04 2024-06-03T05:12:52.385 INFO:teuthology.orchestra.run.smithi038.stdout:(13/32): python3-rgw-19.0.0-2946.g3d89accf.el9. 1.3 MB/s | 99 kB 00:00 2024-06-03T05:12:52.427 INFO:teuthology.orchestra.run.smithi038.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 1.1 MB/s | 107 kB 00:00 2024-06-03T05:12:52.461 INFO:teuthology.orchestra.run.smithi038.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 602 kB/s | 45 kB 00:00 2024-06-03T05:12:52.486 INFO:teuthology.orchestra.run.smithi038.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-06-03T05:12:52.487 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:12:52.512 INFO:teuthology.orchestra.run.smithi038.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-06-03T05:12:52.529 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:12:52.529 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:12:52.536 INFO:teuthology.orchestra.run.smithi038.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 837 kB/s | 42 kB 00:00 2024-06-03T05:12:52.562 INFO:teuthology.orchestra.run.smithi038.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-06-03T05:12:52.587 INFO:teuthology.orchestra.run.smithi038.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-06-03T05:12:52.613 INFO:teuthology.orchestra.run.smithi038.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 475 kB/s | 36 kB 00:00 2024-06-03T05:12:52.638 INFO:teuthology.orchestra.run.smithi038.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-06-03T05:12:52.888 INFO:teuthology.orchestra.run.smithi038.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 104 kB/s | 26 kB 00:00 2024-06-03T05:12:53.102 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:12:53.102 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:12:53.122 INFO:teuthology.orchestra.run.smithi038.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 578 kB/s | 309 kB 00:00 2024-06-03T05:12:53.155 INFO:teuthology.orchestra.run.smithi038.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 182 kB/s | 49 kB 00:00 2024-06-03T05:12:53.256 INFO:teuthology.orchestra.run.smithi038.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-06-03T05:12:53.406 INFO:teuthology.orchestra.run.smithi038.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-06-03T05:12:53.548 INFO:teuthology.orchestra.run.smithi038.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-06-03T05:12:53.582 INFO:teuthology.orchestra.run.smithi038.stdout:(29/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.9 MB/s | 839 kB 00:00 2024-06-03T05:12:53.791 INFO:teuthology.orchestra.run.smithi038.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.1 MB/s | 1.6 MB 00:00 2024-06-03T05:12:53.974 INFO:teuthology.orchestra.run.smithi038.stdout:(31/32): librbd1-19.0.0-2946.g3d89accf.el9.x86_ 8.0 MB/s | 3.1 MB 00:00 2024-06-03T05:12:54.032 INFO:teuthology.orchestra.run.smithi038.stdout:(32/32): librados2-19.0.0-2946.g3d89accf.el9.x8 7.0 MB/s | 3.4 MB 00:00 2024-06-03T05:12:54.034 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:12:54.035 INFO:teuthology.orchestra.run.smithi038.stdout:Total 12 MB/s | 60 MB 00:05 2024-06-03T05:12:54.145 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:12:54.187 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:12:54.187 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:12:54.663 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:12:54.746 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-03T05:12:54.765 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:12:54.766 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:12:54.785 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-03T05:12:54.810 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-03T05:12:54.833 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-03T05:12:54.855 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-03T05:12:55.044 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-03T05:12:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-03T05:12:55.457 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-03T05:12:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 7/34 2024-06-03T05:12:55.656 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 7/34 2024-06-03T05:12:55.744 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 8/34 2024-06-03T05:12:55.771 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/34 2024-06-03T05:12:55.834 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/34 2024-06-03T05:12:55.863 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-03T05:12:55.907 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9. 11/34 2024-06-03T05:12:55.978 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 12/34 2024-06-03T05:12:55.999 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_6 13/34 2024-06-03T05:12:56.063 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_6 13/34 2024-06-03T05:12:56.097 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-03T05:12:56.210 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-03T05:12:56.241 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-03T05:12:56.282 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-03T05:12:56.283 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:12:56.322 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-03T05:12:56.351 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-03T05:12:56.375 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-03T05:12:56.474 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-03T05:12:56.505 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-03T05:12:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-03T05:12:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-03T05:12:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-03T05:12:56.717 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-03T05:12:56.754 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 22/34 2024-06-03T05:12:56.772 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-03T05:12:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-03T05:12:56.846 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 22/34 2024-06-03T05:12:56.876 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 23/34 2024-06-03T05:12:56.920 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-03T05:12:57.177 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-03T05:12:57.298 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 7/34 2024-06-03T05:12:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 7/34 2024-06-03T05:12:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-03T05:12:57.457 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 26/34 2024-06-03T05:12:57.498 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 8/34 2024-06-03T05:12:57.525 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/34 2024-06-03T05:12:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 26/34 2024-06-03T05:12:57.589 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/34 2024-06-03T05:12:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 27/34 2024-06-03T05:12:57.626 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-03T05:12:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x8 28/34 2024-06-03T05:12:57.670 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9. 11/34 2024-06-03T05:12:57.740 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 12/34 2024-06-03T05:12:57.769 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_6 13/34 2024-06-03T05:12:57.834 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_6 13/34 2024-06-03T05:12:57.868 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-03T05:12:57.980 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-03T05:12:58.011 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-03T05:12:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-03T05:12:58.093 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-03T05:12:58.122 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-03T05:12:58.155 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-03T05:12:58.502 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-03T05:12:58.551 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 22/34 2024-06-03T05:12:58.643 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 22/34 2024-06-03T05:12:58.673 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 23/34 2024-06-03T05:12:58.710 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-03T05:12:59.218 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-03T05:12:59.246 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 26/34 2024-06-03T05:12:59.301 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 26/34 2024-06-03T05:12:59.381 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 27/34 2024-06-03T05:12:59.407 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x8 28/34 2024-06-03T05:13:00.040 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 29/34 2024-06-03T05:13:00.089 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 29/34 2024-06-03T05:13:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 29/34 2024-06-03T05:13:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 30/34 2024-06-03T05:13:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 30/34 2024-06-03T05:13:00.784 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-03T05:13:00.785 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-03T05:13:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:00.828 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 31/34 2024-06-03T05:13:01.742 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 29/34 2024-06-03T05:13:01.779 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 29/34 2024-06-03T05:13:02.390 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 29/34 2024-06-03T05:13:02.425 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 30/34 2024-06-03T05:13:02.518 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 30/34 2024-06-03T05:13:02.518 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-03T05:13:02.518 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-03T05:13:02.518 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:02.551 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 31/34 2024-06-03T05:13:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 31/34 2024-06-03T05:13:11.067 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-06-03T05:13:11.067 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-06-03T05:13:11.067 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-06-03T05:13:11.067 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-06-03T05:13:11.067 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-06-03T05:13:11.067 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-06-03T05:13:11.067 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-06-03T05:13:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 32/34 2024-06-03T05:13:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 32/34 2024-06-03T05:13:12.082 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:12.082 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-03T05:13:12.082 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T05:13:12.082 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T05:13:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:12.084 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-03T05:13:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-03T05:13:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-03T05:13:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 31/34 2024-06-03T05:13:12.768 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-06-03T05:13:12.768 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-06-03T05:13:12.768 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-06-03T05:13:12.768 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-06-03T05:13:12.768 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-06-03T05:13:12.768 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-06-03T05:13:12.768 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-06-03T05:13:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:13.385 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-03T05:13:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 1/34 2024-06-03T05:13:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2/34 2024-06-03T05:13:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 3/34 2024-06-03T05:13:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 4/34 2024-06-03T05:13:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 5/34 2024-06-03T05:13:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_6 6/34 2024-06-03T05:13:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 7/34 2024-06-03T05:13:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9. 8/34 2024-06-03T05:13:13.386 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x8 9/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 10/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 11/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 12/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 13/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-03T05:13:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-03T05:13:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-03T05:13:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-03T05:13:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-03T05:13:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-03T05:13:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-03T05:13:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-03T05:13:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-03T05:13:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-03T05:13:13.388 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-03T05:13:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-03T05:13:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 31/34 2024-06-03T05:13:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-03T05:13:13.389 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 33/34 2024-06-03T05:13:13.750 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 32/34 2024-06-03T05:13:13.777 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 32/34 2024-06-03T05:13:13.777 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:13.777 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-03T05:13:13.777 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T05:13:13.777 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T05:13:13.777 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:13.779 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-03T05:13:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-03T05:13:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-03T05:13:14.171 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.172 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-03T05:13:14.173 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:14.174 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:13:14.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-06-03T05:13:15.095 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-03T05:13:15.095 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 1/34 2024-06-03T05:13:15.095 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2/34 2024-06-03T05:13:15.095 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 3/34 2024-06-03T05:13:15.095 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 4/34 2024-06-03T05:13:15.095 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 5/34 2024-06-03T05:13:15.095 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_6 6/34 2024-06-03T05:13:15.095 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 7/34 2024-06-03T05:13:15.095 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9. 8/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x8 9/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 10/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 11/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 12/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 13/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-03T05:13:15.096 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-03T05:13:15.097 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-03T05:13:15.097 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-03T05:13:15.097 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-03T05:13:15.097 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-03T05:13:15.097 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-03T05:13:15.097 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-03T05:13:15.097 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-03T05:13:15.097 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-03T05:13:15.097 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-03T05:13:15.098 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 31/34 2024-06-03T05:13:15.098 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-03T05:13:15.098 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 33/34 2024-06-03T05:13:15.163 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:30 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:13:15.288 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:13:15.289 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:15.289 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:13:15.289 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:15.289 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:13:15.289 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-03T05:13:15.289 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:13:15.289 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-03T05:13:15.290 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-03T05:13:15.290 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-03T05:13:15.290 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:15.290 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:13:15.290 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:15.290 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-06-03T05:13:15.290 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:15.291 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 49 M 2024-06-03T05:13:15.291 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 207 M 2024-06-03T05:13:15.291 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:13:15.627 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 305 kB 00:00 2024-06-03T05:13:15.661 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-06-03T05:13:15.844 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): libcephsqlite-19.0.0-2946.g3d89accf.el9. 435 kB/s | 163 kB 00:00 2024-06-03T05:13:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-03T05:13:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:15.976 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-06-03T05:13:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.976 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:13:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-03T05:13:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-03T05:13:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2024-06-03T05:13:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-03T05:13:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:15.979 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:13:16.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2024-06-03T05:13:16.962 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:30 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:13:17.087 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:13:17.087 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:13:17.087 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:17.087 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:13:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-03T05:13:17.088 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:13:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-03T05:13:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-03T05:13:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-03T05:13:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:17.088 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:13:17.088 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:17.088 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-06-03T05:13:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:17.089 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 49 M 2024-06-03T05:13:17.089 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 207 M 2024-06-03T05:13:17.089 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:13:17.316 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 305 kB 00:00 2024-06-03T05:13:17.350 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-06-03T05:13:17.708 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): libcephsqlite-19.0.0-2946.g3d89accf.el9. 298 kB/s | 163 kB 00:00 2024-06-03T05:13:18.028 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): ceph-test-19.0.0-2946.g3d89accf.el9.x86_ 19 MB/s | 48 MB 00:02 2024-06-03T05:13:18.030 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:18.030 INFO:teuthology.orchestra.run.smithi064.stdout:Total 18 MB/s | 49 MB 00:02 2024-06-03T05:13:18.059 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:13:18.081 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:13:18.081 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:13:18.469 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:13:18.469 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:13:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:13:19.102 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-03T05:13:19.144 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-03T05:13:19.171 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 3/4 2024-06-03T05:13:20.150 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): ceph-test-19.0.0-2946.g3d89accf.el9.x86_ 16 MB/s | 48 MB 00:02 2024-06-03T05:13:20.152 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:20.153 INFO:teuthology.orchestra.run.smithi038.stdout:Total 16 MB/s | 49 MB 00:03 2024-06-03T05:13:20.180 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:13:20.202 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:13:20.203 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:13:20.590 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:13:20.590 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:13:21.133 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:13:21.229 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-03T05:13:21.275 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-03T05:13:21.302 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 3/4 2024-06-03T05:13:24.010 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 3/4 2024-06-03T05:13:24.089 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 4/4 2024-06-03T05:13:26.145 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 3/4 2024-06-03T05:13:26.222 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 4/4 2024-06-03T05:13:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 4/4 2024-06-03T05:13:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 1/4 2024-06-03T05:13:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2/4 2024-06-03T05:13:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-03T05:13:26.749 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-03T05:13:26.749 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:26.749 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:13:26.749 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:26.749 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:26.749 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-03T05:13:26.749 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-03T05:13:26.749 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:26.749 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:13:26.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-06-03T05:13:27.603 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:42 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:13:27.735 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:13:27.738 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:13:27.738 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:27.738 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:13:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-03T05:13:27.738 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:13:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-03T05:13:27.738 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-03T05:13:27.739 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-03T05:13:27.740 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout:Install 36 Packages 2024-06-03T05:13:27.741 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:27.742 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 30 M 2024-06-03T05:13:27.742 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 107 M 2024-06-03T05:13:27.742 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:13:28.733 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 4/4 2024-06-03T05:13:28.733 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 1/4 2024-06-03T05:13:28.733 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2/4 2024-06-03T05:13:28.733 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-03T05:13:28.737 INFO:teuthology.orchestra.run.smithi064.stdout:(1/36): ceph-19.0.0-2946.g3d89accf.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2024-06-03T05:13:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-03T05:13:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:29.026 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:13:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-03T05:13:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-03T05:13:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:29.026 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:13:29.112 INFO:teuthology.orchestra.run.smithi064.stdout:(2/36): ceph-mgr-19.0.0-2946.g3d89accf.el9.x86_ 2.5 MB/s | 1.5 MB 00:00 2024-06-03T05:13:29.163 INFO:teuthology.orchestra.run.smithi064.stdout:(3/36): ceph-mds-19.0.0-2946.g3d89accf.el9.x86_ 3.7 MB/s | 2.4 MB 00:00 2024-06-03T05:13:29.255 INFO:teuthology.orchestra.run.smithi064.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2946.g3d89 2.7 MB/s | 249 kB 00:00 2024-06-03T05:13:29.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2024-06-03T05:13:29.438 INFO:teuthology.orchestra.run.smithi064.stdout:(5/36): ceph-mon-19.0.0-2946.g3d89accf.el9.x86_ 6.7 MB/s | 4.7 MB 00:00 2024-06-03T05:13:29.497 INFO:teuthology.orchestra.run.smithi064.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-06-03T05:13:29.555 INFO:teuthology.orchestra.run.smithi064.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-06-03T05:13:29.614 INFO:teuthology.orchestra.run.smithi064.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-06-03T05:13:29.664 INFO:teuthology.orchestra.run.smithi064.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-06-03T05:13:29.706 INFO:teuthology.orchestra.run.smithi064.stdout:(10/36): python3-cryptography-36.0.1-4.el9.x86_ 4.6 MB/s | 1.2 MB 00:00 2024-06-03T05:13:29.731 INFO:teuthology.orchestra.run.smithi064.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-06-03T05:13:29.823 INFO:teuthology.orchestra.run.smithi064.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-06-03T05:13:29.849 INFO:teuthology.orchestra.run.smithi064.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-06-03T05:13:29.874 INFO:teuthology.orchestra.run.smithi064.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 690 kB/s | 35 kB 00:00 2024-06-03T05:13:29.874 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:43 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:13:29.899 INFO:teuthology.orchestra.run.smithi064.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-06-03T05:13:29.925 INFO:teuthology.orchestra.run.smithi064.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 829 kB/s | 42 kB 00:00 2024-06-03T05:13:30.005 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:13:30.008 INFO:teuthology.orchestra.run.smithi064.stdout:(17/36): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-06-03T05:13:30.008 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:30.008 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:13:30.008 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:30.008 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:13:30.008 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-03T05:13:30.009 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-03T05:13:30.010 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-03T05:13:30.011 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:30.012 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:13:30.012 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:30.012 INFO:teuthology.orchestra.run.smithi038.stdout:Install 36 Packages 2024-06-03T05:13:30.012 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:30.012 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 30 M 2024-06-03T05:13:30.012 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 107 M 2024-06-03T05:13:30.012 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:13:30.042 INFO:teuthology.orchestra.run.smithi064.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-06-03T05:13:30.067 INFO:teuthology.orchestra.run.smithi064.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 737 kB/s | 43 kB 00:00 2024-06-03T05:13:30.117 INFO:teuthology.orchestra.run.smithi064.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 7.0 MB/s | 358 kB 00:00 2024-06-03T05:13:30.151 INFO:teuthology.orchestra.run.smithi064.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 323 kB/s | 11 kB 00:00 2024-06-03T05:13:30.177 INFO:teuthology.orchestra.run.smithi064.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-06-03T05:13:30.193 INFO:teuthology.orchestra.run.smithi064.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-06-03T05:13:30.210 INFO:teuthology.orchestra.run.smithi064.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 692 kB/s | 23 kB 00:00 2024-06-03T05:13:30.227 INFO:teuthology.orchestra.run.smithi064.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 577 kB/s | 19 kB 00:00 2024-06-03T05:13:30.244 INFO:teuthology.orchestra.run.smithi064.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 588 kB/s | 20 kB 00:00 2024-06-03T05:13:30.261 INFO:teuthology.orchestra.run.smithi064.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-06-03T05:13:30.436 INFO:teuthology.orchestra.run.smithi064.stdout:(28/36): ceph-osd-19.0.0-2946.g3d89accf.el9.x86 13 MB/s | 17 MB 00:01 2024-06-03T05:13:30.462 INFO:teuthology.orchestra.run.smithi064.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 363 kB/s | 79 kB 00:00 2024-06-03T05:13:30.487 INFO:teuthology.orchestra.run.smithi064.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-06-03T05:13:30.504 INFO:teuthology.orchestra.run.smithi064.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-06-03T05:13:30.521 INFO:teuthology.orchestra.run.smithi064.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2024-06-03T05:13:30.546 INFO:teuthology.orchestra.run.smithi064.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-06-03T05:13:30.571 INFO:teuthology.orchestra.run.smithi064.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-06-03T05:13:30.596 INFO:teuthology.orchestra.run.smithi064.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-06-03T05:13:30.622 INFO:teuthology.orchestra.run.smithi064.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-06-03T05:13:30.623 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:30.623 INFO:teuthology.orchestra.run.smithi064.stdout:Total 10 MB/s | 30 MB 00:02 2024-06-03T05:13:30.755 INFO:teuthology.orchestra.run.smithi038.stdout:(1/36): ceph-19.0.0-2946.g3d89accf.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-06-03T05:13:30.768 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:13:30.812 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:13:30.813 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:13:31.039 INFO:teuthology.orchestra.run.smithi038.stdout:(2/36): ceph-mgr-19.0.0-2946.g3d89accf.el9.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-06-03T05:13:31.114 INFO:teuthology.orchestra.run.smithi038.stdout:(3/36): ceph-mds-19.0.0-2946.g3d89accf.el9.x86_ 4.5 MB/s | 2.4 MB 00:00 2024-06-03T05:13:31.190 INFO:teuthology.orchestra.run.smithi038.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2946.g3d89 3.2 MB/s | 249 kB 00:00 2024-06-03T05:13:31.298 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:13:31.298 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:13:31.482 INFO:teuthology.orchestra.run.smithi038.stdout:(5/36): ceph-mon-19.0.0-2946.g3d89accf.el9.x86_ 6.5 MB/s | 4.7 MB 00:00 2024-06-03T05:13:31.574 INFO:teuthology.orchestra.run.smithi038.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 660 kB/s | 253 kB 00:00 2024-06-03T05:13:31.624 INFO:teuthology.orchestra.run.smithi038.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-06-03T05:13:31.682 INFO:teuthology.orchestra.run.smithi038.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-06-03T05:13:31.741 INFO:teuthology.orchestra.run.smithi038.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-06-03T05:13:31.800 INFO:teuthology.orchestra.run.smithi038.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-06-03T05:13:31.833 INFO:teuthology.orchestra.run.smithi038.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 3.5 MB/s | 1.2 MB 00:00 2024-06-03T05:13:32.084 INFO:teuthology.orchestra.run.smithi038.stdout:(12/36): ceph-osd-19.0.0-2946.g3d89accf.el9.x86 16 MB/s | 17 MB 00:01 2024-06-03T05:13:32.108 INFO:teuthology.orchestra.run.smithi038.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 610 kB/s | 188 kB 00:00 2024-06-03T05:13:32.150 INFO:teuthology.orchestra.run.smithi038.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 543 kB/s | 172 kB 00:00 2024-06-03T05:13:32.176 INFO:teuthology.orchestra.run.smithi038.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 379 kB/s | 35 kB 00:00 2024-06-03T05:13:32.201 INFO:teuthology.orchestra.run.smithi038.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 552 kB/s | 51 kB 00:00 2024-06-03T05:13:32.226 INFO:teuthology.orchestra.run.smithi038.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-06-03T05:13:32.252 INFO:teuthology.orchestra.run.smithi038.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2024-06-03T05:13:32.527 INFO:teuthology.orchestra.run.smithi038.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-06-03T05:13:32.576 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:13:32.594 INFO:teuthology.orchestra.run.smithi038.stdout:(20/36): luarocks-3.9.2-5.el9.noarch.rpm 386 kB/s | 151 kB 00:00 2024-06-03T05:13:32.633 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-06-03T05:13:32.644 INFO:teuthology.orchestra.run.smithi038.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 438 kB/s | 172 kB 00:00 2024-06-03T05:13:32.669 INFO:teuthology.orchestra.run.smithi038.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-06-03T05:13:32.670 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-06-03T05:13:32.693 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-06-03T05:13:32.720 INFO:teuthology.orchestra.run.smithi038.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 236 kB/s | 18 kB 00:00 2024-06-03T05:13:32.745 INFO:teuthology.orchestra.run.smithi038.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-06-03T05:13:32.761 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-06-03T05:13:32.770 INFO:teuthology.orchestra.run.smithi038.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-06-03T05:13:32.796 INFO:teuthology.orchestra.run.smithi038.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-06-03T05:13:32.810 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-06-03T05:13:32.821 INFO:teuthology.orchestra.run.smithi038.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-06-03T05:13:32.838 INFO:teuthology.orchestra.run.smithi038.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 689 kB/s | 46 kB 00:00 2024-06-03T05:13:32.842 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-06-03T05:13:32.880 INFO:teuthology.orchestra.run.smithi038.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 940 kB/s | 79 kB 00:00 2024-06-03T05:13:32.897 INFO:teuthology.orchestra.run.smithi038.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-06-03T05:13:32.939 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-06-03T05:13:32.955 INFO:teuthology.orchestra.run.smithi038.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-06-03T05:13:32.981 INFO:teuthology.orchestra.run.smithi038.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 429 kB/s | 36 kB 00:00 2024-06-03T05:13:33.006 INFO:teuthology.orchestra.run.smithi038.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 715 kB/s | 90 kB 00:00 2024-06-03T05:13:33.016 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-06-03T05:13:33.040 INFO:teuthology.orchestra.run.smithi038.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-06-03T05:13:33.062 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-06-03T05:13:33.082 INFO:teuthology.orchestra.run.smithi038.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-06-03T05:13:33.097 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-06-03T05:13:33.123 INFO:teuthology.orchestra.run.smithi038.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-06-03T05:13:33.126 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-06-03T05:13:33.128 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:33.128 INFO:teuthology.orchestra.run.smithi038.stdout:Total 9.7 MB/s | 30 MB 00:03 2024-06-03T05:13:33.161 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-06-03T05:13:33.186 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-06-03T05:13:33.226 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-06-03T05:13:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-06-03T05:13:33.276 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:13:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-06-03T05:13:33.320 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:13:33.320 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:13:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-06-03T05:13:33.419 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-06-03T05:13:33.571 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-06-03T05:13:33.653 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-06-03T05:13:33.712 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-06-03T05:13:33.750 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-06-03T05:13:33.776 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:13:33.777 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:13:33.808 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-06-03T05:13:33.872 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-06-03T05:13:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-06-03T05:13:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-06-03T05:13:34.211 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-06-03T05:13:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-06-03T05:13:34.355 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-06-03T05:13:34.480 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-06-03T05:13:34.652 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9. 31/36 2024-06-03T05:13:34.681 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 32/36 2024-06-03T05:13:34.709 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 32/36 2024-06-03T05:13:34.710 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:34.710 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-03T05:13:34.710 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T05:13:34.710 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T05:13:34.710 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:13:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-06-03T05:13:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-06-03T05:13:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-06-03T05:13:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-06-03T05:13:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-06-03T05:13:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-06-03T05:13:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-06-03T05:13:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-06-03T05:13:35.581 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-06-03T05:13:35.615 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-06-03T05:13:35.646 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-06-03T05:13:35.680 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-06-03T05:13:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-06-03T05:13:35.754 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-06-03T05:13:35.797 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-06-03T05:13:35.832 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-06-03T05:13:35.870 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-06-03T05:13:35.946 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-06-03T05:13:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-06-03T05:13:36.181 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-06-03T05:13:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 33/36 2024-06-03T05:13:36.233 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-06-03T05:13:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 33/36 2024-06-03T05:13:36.261 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:36.261 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-03T05:13:36.261 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T05:13:36.261 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T05:13:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:36.279 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-06-03T05:13:36.337 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-06-03T05:13:36.401 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-06-03T05:13:36.597 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-06-03T05:13:36.649 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-06-03T05:13:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 34/36 2024-06-03T05:13:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-06-03T05:13:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 34/36 2024-06-03T05:13:36.749 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:36.749 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-03T05:13:36.750 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T05:13:36.750 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T05:13:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:36.828 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-06-03T05:13:36.864 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-06-03T05:13:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 35/36 2024-06-03T05:13:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-06-03T05:13:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 35/36 2024-06-03T05:13:37.009 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:37.009 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-03T05:13:37.009 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T05:13:37.009 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T05:13:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:37.084 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 36/36 2024-06-03T05:13:37.172 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9. 31/36 2024-06-03T05:13:37.200 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 32/36 2024-06-03T05:13:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 32/36 2024-06-03T05:13:37.228 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:37.228 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-03T05:13:37.228 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T05:13:37.228 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T05:13:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 33/36 2024-06-03T05:13:38.771 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 33/36 2024-06-03T05:13:38.772 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:38.772 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-03T05:13:38.772 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T05:13:38.772 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T05:13:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 36/36 2024-06-03T05:13:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/36 2024-06-03T05:13:38.877 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 3/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 4/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 5/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9. 6/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-06-03T05:13:38.878 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-06-03T05:13:38.879 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-06-03T05:13:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-06-03T05:13:38.880 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-06-03T05:13:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-06-03T05:13:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-06-03T05:13:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-06-03T05:13:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-06-03T05:13:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-06-03T05:13:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-06-03T05:13:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-06-03T05:13:38.881 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-06-03T05:13:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 34/36 2024-06-03T05:13:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 34/36 2024-06-03T05:13:39.268 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:39.268 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-03T05:13:39.269 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T05:13:39.269 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T05:13:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:39.502 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 35/36 2024-06-03T05:13:39.527 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 35/36 2024-06-03T05:13:39.528 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:39.528 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-03T05:13:39.528 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T05:13:39.528 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T05:13:39.528 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 36/36 2024-06-03T05:13:39.962 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-06-03T05:13:39.962 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:39.962 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:13:39.962 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-03T05:13:39.963 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-06-03T05:13:39.964 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:39.965 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:13:40.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-06-03T05:13:40.876 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:55 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:13:40.961 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:13:41.003 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:13:41.004 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:13:41.005 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:13:41.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-06-03T05:13:41.363 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 36/36 2024-06-03T05:13:41.363 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/36 2024-06-03T05:13:41.363 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 3/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 4/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 5/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9. 6/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-06-03T05:13:41.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-06-03T05:13:41.365 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-06-03T05:13:41.366 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-06-03T05:13:41.366 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-06-03T05:13:41.366 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-06-03T05:13:41.366 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-06-03T05:13:41.366 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-06-03T05:13:41.366 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-06-03T05:13:41.366 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-06-03T05:13:41.651 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:56 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:13:41.777 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:13:41.778 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:13:41.778 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:41.778 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:13:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-03T05:13:41.778 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:41.779 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:13:41.779 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:41.779 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:13:41.779 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:41.779 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 763 k 2024-06-03T05:13:41.779 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 769 k 2024-06-03T05:13:41.779 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:13:42.258 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.0.0-2946.g3d89accf.el9.noarch.rpm 1.6 MB/s | 763 kB 00:00 2024-06-03T05:13:42.258 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:42.258 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.6 MB/s | 763 kB 00:00 2024-06-03T05:13:42.259 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:13:42.264 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:13:42.264 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:13:42.270 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:13:42.270 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:13:42.361 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:13:42.441 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-06-03T05:13:42.441 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:42.442 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:13:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-06-03T05:13:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-03T05:13:42.443 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-03T05:13:42.444 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-03T05:13:42.445 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-03T05:13:42.445 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-03T05:13:42.445 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-03T05:13:42.445 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:42.445 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:13:42.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2024-06-03T05:13:42.846 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:13:42.931 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:13:43.289 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:13:43.392 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:57 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:13:43.475 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:13:43.518 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:13:43.520 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:13:43.520 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:13:43.545 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:13:43.545 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:43.545 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:13:43.545 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:43.545 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:43.545 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:13:43.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2024-06-03T05:13:43.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-06-03T05:13:44.168 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:58 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:13:44.297 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:13:44.297 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:13:44.297 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 763 k 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 769 k 2024-06-03T05:13:44.298 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:13:44.340 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:59 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:13:44.467 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:13:44.468 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:13:44.468 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:44.468 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:13:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-03T05:13:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:44.469 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:13:44.469 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:44.469 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:13:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:44.469 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-06-03T05:13:44.469 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 435 k 2024-06-03T05:13:44.469 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:13:44.761 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-19.0.0-2946.g3d89accf.el9.noarch.rpm 1.6 MB/s | 763 kB 00:00 2024-06-03T05:13:44.762 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:44.762 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.6 MB/s | 763 kB 00:00 2024-06-03T05:13:44.762 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:13:44.768 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:13:44.768 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:13:44.774 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:13:44.774 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:13:44.821 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.0.0-2946.g3d89ac 412 kB/s | 145 kB 00:00 2024-06-03T05:13:44.822 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:44.822 INFO:teuthology.orchestra.run.smithi064.stdout:Total 410 kB/s | 145 kB 00:00 2024-06-03T05:13:44.822 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:13:44.830 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:13:44.830 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:13:44.847 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:13:44.890 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:13:44.891 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:13:45.049 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:13:45.105 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf. 1/1 2024-06-03T05:13:45.131 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf. 1/1 2024-06-03T05:13:45.131 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:45.131 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-03T05:13:45.131 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:13:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:13:45.727 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:13:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/1 2024-06-03T05:13:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:45.976 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:13:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:45.976 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:45.976 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:13:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf. 1/1 2024-06-03T05:13:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:46.110 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:13:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:46.111 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:13:46.191 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2024-06-03T05:13:46.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-06-03T05:13:46.794 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:00 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:13:46.920 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:13:46.920 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:13:46.921 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:46.921 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:13:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-03T05:13:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:46.921 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:13:46.921 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:46.921 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:13:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:46.922 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 145 k 2024-06-03T05:13:46.922 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 435 k 2024-06-03T05:13:46.922 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:13:46.943 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:01 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:13:47.027 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:13:47.070 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:13:47.071 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:13:47.071 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:13:47.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-06-03T05:13:47.234 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-19.0.0-2946.g3d89ac 463 kB/s | 145 kB 00:00 2024-06-03T05:13:47.235 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:47.235 INFO:teuthology.orchestra.run.smithi038.stdout:Total 461 kB/s | 145 kB 00:00 2024-06-03T05:13:47.235 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:13:47.243 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:13:47.244 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:13:47.304 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:13:47.304 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:13:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:13:47.527 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf. 1/1 2024-06-03T05:13:47.553 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf. 1/1 2024-06-03T05:13:47.554 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:13:47.554 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-03T05:13:47.554 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:47.726 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:02 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:13:47.853 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-03T05:13:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-06-03T05:13:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:47.857 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 8.6 M 2024-06-03T05:13:47.857 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 103 M 2024-06-03T05:13:47.857 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:13:48.543 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf. 1/1 2024-06-03T05:13:48.543 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:48.543 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:13:48.543 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:13:48.543 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:48.543 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:13:48.591 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2946.g3d 152 kB/s | 32 kB 00:00 2024-06-03T05:13:48.691 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2946.g3d8 53 kB/s | 16 kB 00:00 2024-06-03T05:13:48.766 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-06-03T05:13:48.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2024-06-03T05:13:48.833 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-06-03T05:13:49.034 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-06-03T05:13:49.268 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-06-03T05:13:49.292 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-06-03T05:13:49.360 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): ceph-mgr-dashboard-19.0.0-2946.g3d89acc 3.6 MB/s | 3.5 MB 00:00 2024-06-03T05:13:49.381 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:03 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:13:49.385 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.5 MB/s | 144 kB 00:00 2024-06-03T05:13:49.465 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:13:49.469 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-06-03T05:13:49.508 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:13:49.509 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:13:49.509 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:13:49.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2024-06-03T05:13:49.611 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 123 kB/s | 31 kB 00:00 2024-06-03T05:13:49.694 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.8 MB/s | 2.0 MB 00:00 2024-06-03T05:13:49.698 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:49.698 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.7 MB/s | 8.6 MB 00:01 2024-06-03T05:13:49.747 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:13:49.765 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:13:49.765 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:13:49.958 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:13:49.958 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:13:50.173 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:04 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:13:50.299 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:13:50.300 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:50.300 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-03T05:13:50.301 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-03T05:13:50.302 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-03T05:13:50.302 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-03T05:13:50.302 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-03T05:13:50.302 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-03T05:13:50.302 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:50.302 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:13:50.302 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:13:50.302 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-06-03T05:13:50.302 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:50.303 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 8.6 M 2024-06-03T05:13:50.303 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 103 M 2024-06-03T05:13:50.303 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:13:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:13:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-06-03T05:13:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-06-03T05:13:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-06-03T05:13:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-06-03T05:13:50.797 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-06-03T05:13:50.964 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2946.g3d 157 kB/s | 32 kB 00:00 2024-06-03T05:13:51.064 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2946.g3d8 54 kB/s | 16 kB 00:00 2024-06-03T05:13:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-06-03T05:13:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-06-03T05:13:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-06-03T05:13:51.215 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 269 kB 00:00 2024-06-03T05:13:51.240 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-06-03T05:13:51.257 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-06-03T05:13:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9 10/12 2024-06-03T05:13:51.357 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): ceph-mgr-dashboard-19.0.0-2946.g3d89acc 5.9 MB/s | 3.5 MB 00:00 2024-06-03T05:13:51.399 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-06-03T05:13:51.432 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.1 MB/s | 548 kB 00:00 2024-06-03T05:13:51.466 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-06-03T05:13:51.491 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-06-03T05:13:51.525 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.7 MB/s | 863 kB 00:00 2024-06-03T05:13:51.550 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-06-03T05:13:51.642 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.4 MB/s | 2.0 MB 00:00 2024-06-03T05:13:51.644 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:51.645 INFO:teuthology.orchestra.run.smithi038.stdout:Total 6.4 MB/s | 8.6 MB 00:01 2024-06-03T05:13:51.696 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:13:51.714 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:13:51.714 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:13:51.904 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:13:51.904 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:13:52.372 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el 11/12 2024-06-03T05:13:52.406 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noa 12/12 2024-06-03T05:13:52.456 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:13:52.527 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-06-03T05:13:52.562 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-06-03T05:13:52.610 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-06-03T05:13:52.638 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-06-03T05:13:52.768 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-06-03T05:13:52.906 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-06-03T05:13:53.006 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-06-03T05:13:53.054 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-06-03T05:13:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-06-03T05:13:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9 10/12 2024-06-03T05:13:53.652 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noa 12/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el 1/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noa 2/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9 3/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-06-03T05:13:53.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-03T05:13:53.960 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-03T05:13:53.961 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-03T05:13:53.961 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-03T05:13:53.961 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-03T05:13:53.961 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-03T05:13:53.961 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:53.961 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:13:54.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-03T05:13:54.229 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el 11/12 2024-06-03T05:13:54.263 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noa 12/12 2024-06-03T05:13:54.799 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:09 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:13:54.928 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:13:54.929 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-03T05:13:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-03T05:13:54.931 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-03T05:13:54.931 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-03T05:13:54.931 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-03T05:13:54.931 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-03T05:13:54.931 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:54.931 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:13:54.931 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-06-03T05:13:54.931 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-06-03T05:13:54.931 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:13:54.932 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 38 M 2024-06-03T05:13:54.932 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 209 M 2024-06-03T05:13:54.932 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:13:55.368 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-06-03T05:13:55.435 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-06-03T05:13:55.452 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-06-03T05:13:55.477 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 728 kB/s | 18 kB 00:00 2024-06-03T05:13:55.511 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 35 kB 00:00 2024-06-03T05:13:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noa 12/12 2024-06-03T05:13:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el 1/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noa 2/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9 3/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-06-03T05:13:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-06-03T05:13:55.628 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-06-03T05:13:55.670 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 5.8 MB/s | 244 kB 00:00 2024-06-03T05:13:55.738 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-06-03T05:13:55.779 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-06-03T05:13:55.862 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 5.1 MB 00:00 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-03T05:13:55.926 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-03T05:13:55.927 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-03T05:13:55.927 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-03T05:13:55.927 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-03T05:13:55.927 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-03T05:13:55.927 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-03T05:13:55.927 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:55.927 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:13:55.979 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-2 9.2 MB/s | 7.4 MB 00:00 2024-06-03T05:13:56.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-03T05:13:56.197 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-06-03T05:13:56.198 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:56.198 INFO:teuthology.orchestra.run.smithi064.stdout:Total 30 MB/s | 38 MB 00:01 2024-06-03T05:13:56.559 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:13:56.587 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:13:56.587 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:13:56.778 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:10 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:13:56.911 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:13:56.912 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-06-03T05:13:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:13:56.912 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-06-03T05:13:56.912 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:13:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-03T05:13:56.912 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-03T05:13:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:56.914 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:13:56.914 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-06-03T05:13:56.914 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-06-03T05:13:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:13:56.914 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 38 M 2024-06-03T05:13:56.914 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 209 M 2024-06-03T05:13:56.915 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:13:57.041 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:13:57.042 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:13:57.293 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-06-03T05:13:57.376 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-06-03T05:13:57.401 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-06-03T05:13:57.485 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 218 kB/s | 18 kB 00:00 2024-06-03T05:13:57.552 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 35 kB 00:00 2024-06-03T05:13:57.661 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-06-03T05:13:57.711 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-06-03T05:13:57.827 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:13:57.870 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-06-03T05:13:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-03T05:13:57.945 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 5.1 MB 00:00 2024-06-03T05:13:57.953 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-03T05:13:57.970 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-06-03T05:13:58.057 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-03T05:13:58.095 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-03T05:13:58.096 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-2 7.5 MB/s | 7.4 MB 00:00 2024-06-03T05:13:58.279 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-03T05:13:58.311 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-03T05:13:58.390 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-03T05:13:59.075 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-03T05:13:59.146 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 14 MB/s | 16 MB 00:01 2024-06-03T05:13:59.147 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:13:59.147 INFO:teuthology.orchestra.run.smithi038.stdout:Total 17 MB/s | 38 MB 00:02 2024-06-03T05:13:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-03T05:13:59.517 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:13:59.544 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:13:59.545 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:00.020 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:00.020 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:00.517 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T05:14:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-03T05:14:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-03T05:14:01.032 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-03T05:14:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-03T05:14:01.254 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-03T05:14:01.286 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-03T05:14:01.365 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-03T05:14:01.597 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-03T05:14:01.626 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89a 12/12 2024-06-03T05:14:02.068 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-03T05:14:02.188 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-03T05:14:03.514 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T05:14:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89a 12/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89a 1/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T05:14:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-03T05:14:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-03T05:14:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:04.487 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:04.488 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:04.607 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-03T05:14:04.640 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89a 12/12 2024-06-03T05:14:04.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-06-03T05:14:05.467 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:20 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:05.596 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:05.597 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:14:05.597 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:05.597 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-03T05:14:05.597 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:14:05.597 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-03T05:14:05.598 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:05.599 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:05.599 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:05.599 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-06-03T05:14:05.599 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:05.599 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.2 M 2024-06-03T05:14:05.599 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-06-03T05:14:05.599 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:06.067 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 289 kB/s | 26 kB 00:00 2024-06-03T05:14:06.092 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 168 kB/s | 19 kB 00:00 2024-06-03T05:14:06.159 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.4 MB/s | 159 kB 00:00 2024-06-03T05:14:06.185 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-06-03T05:14:06.210 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 5.4 MB/s | 279 kB 00:00 2024-06-03T05:14:06.235 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-06-03T05:14:06.260 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): ceph-mgr-rook-19.0.0-2946.g3d89accf.el9 174 kB/s | 49 kB 00:00 2024-06-03T05:14:06.494 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-06-03T05:14:06.521 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-06-03T05:14:06.661 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 513 kB/s | 206 kB 00:00 2024-06-03T05:14:06.686 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-06-03T05:14:06.786 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-06-03T05:14:06.811 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.2 MB/s | 1.0 MB 00:00 2024-06-03T05:14:06.815 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:06.815 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-06-03T05:14:06.882 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:06.898 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:06.898 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:06.992 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:06.992 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:07.156 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89a 12/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89a 1/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T05:14:07.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-03T05:14:07.522 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-03T05:14:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:07.552 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:14:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-03T05:14:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-03T05:14:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-03T05:14:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-03T05:14:07.552 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-03T05:14:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-03T05:14:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-03T05:14:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-03T05:14:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-03T05:14:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-03T05:14:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-03T05:14:07.553 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:07.553 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-03T05:14:07.773 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-03T05:14:07.829 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-03T05:14:07.865 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-03T05:14:07.906 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-03T05:14:07.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2024-06-03T05:14:08.004 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-03T05:14:08.117 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-03T05:14:08.175 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-03T05:14:08.539 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:22 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:08.669 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:08.670 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:08.670 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:14:08.670 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:08.670 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:08.670 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-03T05:14:08.670 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:14:08.670 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:08.671 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:08.672 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2024-06-03T05:14:08.672 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:08.672 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.2 M 2024-06-03T05:14:08.672 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 26 M 2024-06-03T05:14:08.672 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:08.936 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-03T05:14:08.940 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 297 kB/s | 26 kB 00:00 2024-06-03T05:14:08.965 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 172 kB/s | 19 kB 00:00 2024-06-03T05:14:08.980 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-03T05:14:09.014 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-03T05:14:09.063 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-03T05:14:09.065 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-06-03T05:14:09.091 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-06-03T05:14:09.091 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 13/13 2024-06-03T05:14:09.133 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 279 kB 00:00 2024-06-03T05:14:09.158 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): ceph-mgr-rook-19.0.0-2946.g3d89accf.el9 162 kB/s | 49 kB 00:00 2024-06-03T05:14:09.183 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 582 kB/s | 54 kB 00:00 2024-06-03T05:14:09.275 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-06-03T05:14:09.300 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-06-03T05:14:09.392 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 984 kB/s | 206 kB 00:00 2024-06-03T05:14:09.417 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-06-03T05:14:09.451 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-06-03T05:14:09.535 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-06-03T05:14:09.536 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:09.536 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-06-03T05:14:09.610 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:09.626 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:09.627 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:09.722 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:09.723 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:09.796 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 13/13 2024-06-03T05:14:09.796 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 1/13 2024-06-03T05:14:09.796 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-03T05:14:09.797 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-03T05:14:10.123 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-03T05:14:10.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:10.124 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:10.124 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:14:10.124 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-03T05:14:10.124 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-03T05:14:10.124 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-03T05:14:10.124 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-03T05:14:10.124 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-03T05:14:10.124 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-03T05:14:10.125 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-03T05:14:10.125 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-03T05:14:10.125 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-03T05:14:10.125 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-03T05:14:10.125 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-03T05:14:10.125 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-03T05:14:10.125 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:10.125 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:10.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-06-03T05:14:10.416 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-03T05:14:10.488 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-03T05:14:10.552 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-03T05:14:10.587 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-03T05:14:10.630 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-03T05:14:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-03T05:14:10.838 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-03T05:14:10.890 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-03T05:14:10.981 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:25 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:11.110 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:11.111 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:14:11.111 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:11.111 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-03T05:14:11.111 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:14:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-03T05:14:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-03T05:14:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-03T05:14:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-03T05:14:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-03T05:14:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:11.112 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:11.112 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:11.112 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-06-03T05:14:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:11.113 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 7.0 M 2024-06-03T05:14:11.113 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-06-03T05:14:11.113 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:11.498 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-06-03T05:14:11.565 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2946.g3d89accf.e 634 kB/s | 142 kB 00:00 2024-06-03T05:14:11.657 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-03T05:14:11.673 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-06-03T05:14:11.698 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 432 kB/s | 58 kB 00:00 2024-06-03T05:14:11.719 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-03T05:14:11.724 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.4 MB/s | 548 kB 00:00 2024-06-03T05:14:11.749 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 678 kB/s | 51 kB 00:00 2024-06-03T05:14:11.749 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:11.750 INFO:teuthology.orchestra.run.smithi064.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-06-03T05:14:11.753 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-03T05:14:11.802 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-03T05:14:11.823 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:11.831 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 13/13 2024-06-03T05:14:11.835 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:11.835 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:11.940 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:11.940 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-03T05:14:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-03T05:14:12.617 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 13/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 1/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-03T05:14:12.618 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-03T05:14:12.619 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-03T05:14:12.619 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-03T05:14:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-03T05:14:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-03T05:14:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-03T05:14:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-03T05:14:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-03T05:14:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:12.984 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-03T05:14:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-03T05:14:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-03T05:14:13.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2024-06-03T05:14:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 6/6 2024-06-03T05:14:13.823 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:27 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:13.953 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:13.953 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:13.954 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:13.955 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:13.955 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-06-03T05:14:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:13.955 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 7.0 M 2024-06-03T05:14:13.955 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 33 M 2024-06-03T05:14:13.955 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 6/6 2024-06-03T05:14:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/6 2024-06-03T05:14:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-03T05:14:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-03T05:14:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-03T05:14:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-03T05:14:14.263 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-03T05:14:14.264 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:14.264 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:14.264 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:14:14.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-03T05:14:14.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-03T05:14:14.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-03T05:14:14.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-03T05:14:14.264 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-03T05:14:14.264 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:14.265 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:14.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-06-03T05:14:14.503 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-06-03T05:14:14.637 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 6.0 MB 00:00 2024-06-03T05:14:14.737 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2946.g3d89accf.e 365 kB/s | 142 kB 00:00 2024-06-03T05:14:15.021 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-06-03T05:14:15.062 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 170 kB/s | 51 kB 00:00 2024-06-03T05:14:15.087 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 938 kB/s | 548 kB 00:00 2024-06-03T05:14:15.090 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:15.091 INFO:teuthology.orchestra.run.smithi038.stdout:Total 6.1 MB/s | 7.0 MB 00:01 2024-06-03T05:14:15.091 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:30 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:15.165 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:15.177 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:15.177 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:15.220 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:15.220 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:15.220 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:14:15.220 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:15.220 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-03T05:14:15.221 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:14:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-03T05:14:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:15.221 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:15.221 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:15.221 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-06-03T05:14:15.221 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:15.222 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 970 k 2024-06-03T05:14:15.222 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.8 M 2024-06-03T05:14:15.222 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:15.281 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:15.281 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:15.442 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 437 kB/s | 80 kB 00:00 2024-06-03T05:14:15.576 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.0.0-2946.g3d89accf.el9.x86_ 2.7 MB/s | 890 kB 00:00 2024-06-03T05:14:15.576 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:15.577 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.7 MB/s | 970 kB 00:00 2024-06-03T05:14:15.588 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:15.602 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:15.612 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:15.613 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:15.695 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:15.696 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-03T05:14:15.833 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-03T05:14:15.889 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:16.032 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-03T05:14:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2/2 2024-06-03T05:14:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-03T05:14:16.471 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-03T05:14:16.549 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-03T05:14:16.577 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 6/6 2024-06-03T05:14:16.716 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2/2 2024-06-03T05:14:16.717 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/2 2024-06-03T05:14:16.982 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-03T05:14:16.982 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:16.982 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:16.982 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-03T05:14:16.982 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:16.982 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:17.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-06-03T05:14:17.383 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 6/6 2024-06-03T05:14:17.383 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 1/6 2024-06-03T05:14:17.383 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-03T05:14:17.383 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-03T05:14:17.384 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-03T05:14:17.384 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-03T05:14:17.673 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-03T05:14:17.673 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:17.673 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:17.673 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:14:17.673 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-03T05:14:17.674 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-03T05:14:17.674 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-03T05:14:17.674 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-03T05:14:17.674 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-03T05:14:17.674 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:17.675 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:17.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2024-06-03T05:14:17.899 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:32 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:18.028 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:18.028 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:18.028 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-06-03T05:14:18.028 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:18.029 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:18.029 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-03T05:14:18.029 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:14:18.029 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-03T05:14:18.029 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:18.029 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:18.029 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:18.029 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-06-03T05:14:18.029 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:18.030 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 341 k 2024-06-03T05:14:18.030 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.5 M 2024-06-03T05:14:18.030 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:18.164 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 798 kB/s | 77 kB 00:00 2024-06-03T05:14:18.339 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-volume-19.0.0-2946.g3d89accf.el9.no 970 kB/s | 264 kB 00:00 2024-06-03T05:14:18.339 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:18.339 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.1 MB/s | 341 kB 00:00 2024-06-03T05:14:18.351 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:18.362 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:18.362 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:18.397 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:18.397 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:18.501 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:32 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:18.548 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:18.629 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:18.630 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:18.630 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:14:18.630 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:18.630 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:18.630 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-03T05:14:18.630 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:14:18.630 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-03T05:14:18.630 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:18.630 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:18.631 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:18.631 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-06-03T05:14:18.631 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:18.631 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 970 k 2024-06-03T05:14:18.631 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.8 M 2024-06-03T05:14:18.631 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:18.765 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-06-03T05:14:18.790 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 865 kB/s | 80 kB 00:00 2024-06-03T05:14:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2/2 2024-06-03T05:14:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2/2 2024-06-03T05:14:18.818 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:14:18.818 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-03T05:14:18.818 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:19.006 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-fuse-19.0.0-2946.g3d89accf.el9.x86_ 2.8 MB/s | 890 kB 00:00 2024-06-03T05:14:19.007 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:19.007 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.5 MB/s | 970 kB 00:00 2024-06-03T05:14:19.022 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:19.043 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:19.043 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:19.127 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:19.127 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:19.327 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:19.471 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-03T05:14:19.476 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 1/2 2024-06-03T05:14:19.541 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2/2 2024-06-03T05:14:19.764 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-06-03T05:14:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:19.765 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:14:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-03T05:14:19.765 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:19.765 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:20.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-06-03T05:14:20.216 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2/2 2024-06-03T05:14:20.216 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/2 2024-06-03T05:14:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-03T05:14:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:20.480 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-03T05:14:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:20.480 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:20.675 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:35 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:20.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-volume 2024-06-03T05:14:20.805 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:20.805 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:14:20.805 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:20.805 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-03T05:14:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:20.806 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:20.806 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:20.806 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:14:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:20.806 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 127 k 2024-06-03T05:14:20.806 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-06-03T05:14:20.807 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:21.037 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.0.0-2946.g3d89accf.el9.x86_64 550 kB/s | 127 kB 00:00 2024-06-03T05:14:21.038 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:21.038 INFO:teuthology.orchestra.run.smithi064.stdout:Total 546 kB/s | 127 kB 00:00 2024-06-03T05:14:21.038 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:21.045 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:21.045 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:21.112 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:21.112 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:21.274 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:21.353 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:35 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:21.388 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:21.484 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:21.484 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-06-03T05:14:21.485 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:21.486 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 341 k 2024-06-03T05:14:21.486 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.5 M 2024-06-03T05:14:21.486 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:21.861 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 767 kB/s | 77 kB 00:00 2024-06-03T05:14:21.910 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:22.028 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-volume-19.0.0-2946.g3d89accf.el9.no 986 kB/s | 264 kB 00:00 2024-06-03T05:14:22.029 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:22.029 INFO:teuthology.orchestra.run.smithi038.stdout:Total 627 kB/s | 341 kB 00:00 2024-06-03T05:14:22.041 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:22.052 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:22.052 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:22.092 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:22.092 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:22.188 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:22.188 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:22.188 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:22.245 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-06-03T05:14:22.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-06-03T05:14:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2/2 2024-06-03T05:14:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2/2 2024-06-03T05:14:22.489 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:14:22.489 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-03T05:14:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:23.075 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:23.159 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:23.204 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:23.205 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:14:23.205 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:23.220 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 1/2 2024-06-03T05:14:23.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-06-03T05:14:23.538 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-06-03T05:14:23.538 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:23.538 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:23.538 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-03T05:14:23.538 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-03T05:14:23.538 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:23.539 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:23.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2024-06-03T05:14:23.874 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:24.003 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:24.004 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:24.004 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:14:24.004 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:24.004 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:24.004 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-03T05:14:24.004 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:24.004 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:24.004 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:24.005 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:14:24.005 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:24.005 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 32 k 2024-06-03T05:14:24.005 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 142 k 2024-06-03T05:14:24.005 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:24.224 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.0.0-2946.g3d89accf.el9.x86_6 145 kB/s | 32 kB 00:00 2024-06-03T05:14:24.224 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:24.224 INFO:teuthology.orchestra.run.smithi064.stdout:Total 144 kB/s | 32 kB 00:00 2024-06-03T05:14:24.225 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:24.228 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:24.228 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:24.237 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:24.238 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:24.323 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:24.481 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:38 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:24.610 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:24.610 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:24.610 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:14:24.610 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 127 k 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 456 k 2024-06-03T05:14:24.611 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:24.817 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-19.0.0-2946.g3d89accf.el9.x86_64 615 kB/s | 127 kB 00:00 2024-06-03T05:14:24.818 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:24.818 INFO:teuthology.orchestra.run.smithi038.stdout:Total 611 kB/s | 127 kB 00:00 2024-06-03T05:14:24.818 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:24.824 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:24.824 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:24.892 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:24.892 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:24.914 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:25.051 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:25.161 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:25.237 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:25.237 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:25.237 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:25.237 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:25.237 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:25.237 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:25.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-06-03T05:14:25.585 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:25.837 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:25.837 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:25.837 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:25.838 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:25.838 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:25.838 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:26.014 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:41 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:26.099 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:26.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2024-06-03T05:14:26.146 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:26.147 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:14:26.147 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:26.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-06-03T05:14:26.708 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:40 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:26.793 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:26.811 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:41 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:26.839 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:26.840 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:14:26.840 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:26.896 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:26.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2024-06-03T05:14:26.940 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:26.941 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:14:26.941 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:27.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-06-03T05:14:27.527 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:41 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:27.619 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:42 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:27.658 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:27.658 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:27.658 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:14:27.658 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:27.659 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:27.659 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-03T05:14:27.659 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:27.659 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:27.659 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:27.659 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:14:27.659 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:27.659 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 32 k 2024-06-03T05:14:27.660 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 142 k 2024-06-03T05:14:27.660 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:27.702 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:27.747 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:27.748 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:14:27.748 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:27.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-06-03T05:14:27.824 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-19.0.0-2946.g3d89accf.el9.x86_6 193 kB/s | 32 kB 00:00 2024-06-03T05:14:27.825 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:27.825 INFO:teuthology.orchestra.run.smithi038.stdout:Total 191 kB/s | 32 kB 00:00 2024-06-03T05:14:27.825 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:27.852 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:27.852 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:27.936 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:28.044 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:28.422 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:43 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:28.468 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:28.506 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:28.550 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:28.551 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:14:28.552 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:28.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-06-03T05:14:28.695 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:28.695 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:28.695 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:28.695 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:28.695 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:28.695 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:28.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2024-06-03T05:14:29.235 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:29.320 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:29.364 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:29.365 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:14:29.365 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:29.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-06-03T05:14:29.494 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:43 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:29.577 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:29.622 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:29.623 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:14:29.623 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:29.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2024-06-03T05:14:30.051 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:45 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:30.134 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:30.179 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:30.180 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:14:30.180 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:30.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-06-03T05:14:30.280 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:44 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:30.364 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:30.409 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:30.410 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:14:30.410 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:30.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2024-06-03T05:14:30.855 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:45 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:30.985 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:30.986 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:30.986 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-06-03T05:14:30.986 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:30.986 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:30.986 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-03T05:14:30.986 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:30.986 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:30.987 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:30.987 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:14:30.987 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:30.987 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-06-03T05:14:30.987 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-06-03T05:14:30.987 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:31.077 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:45 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:31.161 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:31.202 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.0.0-2946.g3d89accf.el9.x86_64.rpm 398 kB/s | 85 kB 00:00 2024-06-03T05:14:31.202 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:31.202 INFO:teuthology.orchestra.run.smithi064.stdout:Total 395 kB/s | 85 kB 00:00 2024-06-03T05:14:31.202 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:31.206 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:31.207 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:14:31.207 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:31.210 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:31.210 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:31.274 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2024-06-03T05:14:31.280 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:31.281 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:31.432 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:31.544 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:31.871 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:45 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:31.954 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:31.999 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:32.000 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:14:32.000 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:32.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2024-06-03T05:14:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:32.163 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:32.164 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:32.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-06-03T05:14:32.669 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:46 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:32.752 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:32.796 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:32.797 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:14:32.797 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:32.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2024-06-03T05:14:33.004 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:48 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:33.136 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:14:33.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:33.138 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.1 M 2024-06-03T05:14:33.138 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 13 M 2024-06-03T05:14:33.138 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:33.454 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:47 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:33.536 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:14:33.581 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:33.582 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:14:33.583 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:33.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2024-06-03T05:14:33.704 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.0.0-2946.g3d89accf.el9.x86_64.rpm 5.5 MB/s | 3.1 MB 00:00 2024-06-03T05:14:33.705 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:33.705 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.4 MB/s | 3.1 MB 00:00 2024-06-03T05:14:33.705 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:33.716 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:33.716 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:33.801 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:33.801 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:34.254 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:48 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:34.350 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:34.350 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:14:34.350 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-03T05:14:34.350 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T05:14:34.350 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T05:14:34.350 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:34.384 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:34.384 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 85 k 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 231 k 2024-06-03T05:14:34.385 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:34.696 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-19.0.0-2946.g3d89accf.el9.x86_64.rpm 274 kB/s | 85 kB 00:00 2024-06-03T05:14:34.696 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:34.697 INFO:teuthology.orchestra.run.smithi038.stdout:Total 273 kB/s | 85 kB 00:00 2024-06-03T05:14:34.697 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:34.705 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:34.705 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:34.771 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:34.772 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:35.420 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:35.420 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:35.629 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:35.629 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:35.629 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:35.630 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:35.630 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:35.630 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:35.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-06-03T05:14:35.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2024-06-03T05:14:36.265 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:51 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:36.393 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:36.393 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:36.393 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-06-03T05:14:36.394 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:36.394 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:36.394 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-03T05:14:36.394 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:36.394 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:36.394 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:36.394 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:14:36.394 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:36.395 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 171 k 2024-06-03T05:14:36.395 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 494 k 2024-06-03T05:14:36.395 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:36.477 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:50 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:36.606 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:36.607 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:36.607 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:14:36.607 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:36.607 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:36.607 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-03T05:14:36.607 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:36.607 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:36.607 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:36.608 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:14:36.608 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:36.608 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.1 M 2024-06-03T05:14:36.608 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 13 M 2024-06-03T05:14:36.608 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:36.735 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.0.0-2946.g3d89accf.el9.x86_64.rpm 502 kB/s | 171 kB 00:00 2024-06-03T05:14:36.736 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:36.736 INFO:teuthology.orchestra.run.smithi064.stdout:Total 500 kB/s | 171 kB 00:00 2024-06-03T05:14:36.736 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:36.745 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:36.745 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:36.812 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:36.812 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:37.087 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:37.161 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-19.0.0-2946.g3d89accf.el9.x86_64.rpm 5.6 MB/s | 3.1 MB 00:00 2024-06-03T05:14:37.162 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:37.162 INFO:teuthology.orchestra.run.smithi038.stdout:Total 5.6 MB/s | 3.1 MB 00:00 2024-06-03T05:14:37.162 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:37.172 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:37.172 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:37.260 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:37.260 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:37.488 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:37.746 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:37.746 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:37.746 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:37.746 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:37.746 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:37.746 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:37.805 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T05:14:37.805 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-03T05:14:37.805 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T05:14:37.805 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T05:14:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:38.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base-debuginfo 2024-06-03T05:14:38.609 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:53 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:38.742 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:38.742 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 69 M 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 23 M 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 92 M 2024-06-03T05:14:38.743 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 503 M 2024-06-03T05:14:38.744 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:38.911 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:38.911 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:39.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2024-06-03T05:14:39.749 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:53 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:39.878 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:39.879 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:39.879 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-06-03T05:14:39.879 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 171 k 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 494 k 2024-06-03T05:14:39.880 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:40.784 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-19.0.0-2946.g3d89accf.el9.x86_64.rpm 189 kB/s | 171 kB 00:00 2024-06-03T05:14:40.785 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:40.785 INFO:teuthology.orchestra.run.smithi038.stdout:Total 188 kB/s | 171 kB 00:00 2024-06-03T05:14:40.785 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:40.796 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:40.796 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:40.868 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:40.868 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:40.991 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): ceph-debuginfo-19.0.0-2946.g3d89accf.el9 10 MB/s | 23 MB 00:02 2024-06-03T05:14:41.040 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:41.143 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:41.551 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:14:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:41.817 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:14:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:41.817 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:14:42.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base-debuginfo 2024-06-03T05:14:42.656 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:56 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:14:42.785 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 69 M 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 23 M 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:14:42.786 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:14:42.787 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-06-03T05:14:42.787 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:14:42.787 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 92 M 2024-06-03T05:14:42.787 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 503 M 2024-06-03T05:14:42.787 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:14:43.150 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-base-debuginfo-19.0.0-2946.g3d89acc 16 MB/s | 69 MB 00:04 2024-06-03T05:14:43.152 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:43.152 INFO:teuthology.orchestra.run.smithi064.stdout:Total 21 MB/s | 92 MB 00:04 2024-06-03T05:14:43.153 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:14:43.158 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:14:43.158 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:14:43.776 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:14:43.776 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:14:46.374 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): ceph-debuginfo-19.0.0-2946.g3d89accf.el9 6.5 MB/s | 23 MB 00:03 2024-06-03T05:14:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:14:48.374 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-base-debuginfo-19.0.0-2946.g3d89acc 12 MB/s | 69 MB 00:05 2024-06-03T05:14:48.376 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:14:48.377 INFO:teuthology.orchestra.run.smithi038.stdout:Total 16 MB/s | 92 MB 00:05 2024-06-03T05:14:48.377 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:14:48.382 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:14:48.382 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:14:49.004 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:14:49.005 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:14:52.376 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:14:54.725 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/2 2024-06-03T05:14:54.799 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 2/2 2024-06-03T05:14:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 2/2 2024-06-03T05:14:58.182 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/2 2024-06-03T05:14:58.417 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2/2 2024-06-03T05:14:58.417 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:58.417 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:14:58.418 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:58.418 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:14:58.418 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:58.418 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:14:58.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-common-debuginfo 2024-06-03T05:14:59.261 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:14 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:14:59.390 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:14:59.391 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:14:59.391 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:59.391 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:14:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 451 M 2024-06-03T05:14:59.391 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:59.391 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:14:59.392 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:14:59.392 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:14:59.392 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:14:59.392 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 451 M 2024-06-03T05:14:59.392 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.8 G 2024-06-03T05:14:59.392 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:14:59.938 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/2 2024-06-03T05:15:00.006 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 2/2 2024-06-03T05:15:03.423 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 2/2 2024-06-03T05:15:03.423 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/2 2024-06-03T05:15:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2/2 2024-06-03T05:15:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:15:03.658 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:15:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:15:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:15:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:15:03.658 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:15:03.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-common-debuginfo 2024-06-03T05:15:04.470 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:18 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:15:04.603 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:15:04.603 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:15:04.603 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:15:04.603 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:15:04.603 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:15:04.603 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 451 M 2024-06-03T05:15:04.603 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:15:04.604 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:15:04.604 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:15:04.604 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:15:04.604 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:15:04.604 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 451 M 2024-06-03T05:15:04.604 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.8 G 2024-06-03T05:15:04.604 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:15:22.376 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-common-debuginfo-19.0.0-2946.g3d89accf.el9 20 MB/s | 451 MB 00:22 2024-06-03T05:15:22.377 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:15:22.377 INFO:teuthology.orchestra.run.smithi064.stdout:Total 20 MB/s | 451 MB 00:22 2024-06-03T05:15:22.378 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:15:22.382 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:15:22.382 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:15:25.190 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:15:25.190 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:15:35.282 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-common-debuginfo-19.0.0-2946.g3d89accf.el9 15 MB/s | 451 MB 00:30 2024-06-03T05:15:35.283 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:15:35.283 INFO:teuthology.orchestra.run.smithi038.stdout:Total 15 MB/s | 451 MB 00:30 2024-06-03T05:15:35.284 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:15:35.288 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:15:35.288 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:15:38.097 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:15:38.097 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:16:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:16:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:16:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:16:21.254 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:16:21.254 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:16:21.255 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:16:21.255 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:16:21.255 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:16:21.255 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:16:21.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-06-03T05:16:22.069 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:37 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:16:22.200 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:16:22.201 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.5 M 2024-06-03T05:16:22.202 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 6.6 M 2024-06-03T05:16:22.202 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:16:22.709 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-29 2.9 MB/s | 1.5 MB 00:00 2024-06-03T05:16:22.710 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:16:22.710 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.9 MB/s | 1.5 MB 00:00 2024-06-03T05:16:22.710 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:16:22.714 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:16:22.714 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:16:22.761 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:16:22.761 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:16:23.037 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:16:23.144 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2946. 1/1 2024-06-03T05:16:23.517 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-2946. 1/1 2024-06-03T05:16:23.747 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2946. 1/1 2024-06-03T05:16:23.747 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:16:23.747 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:16:23.747 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:16:23.747 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:16:23.747 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:16:23.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw-debuginfo 2024-06-03T05:16:24.505 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:39 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:16:24.633 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:16:24.634 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:16:24.634 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 187 M 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 187 M 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 797 M 2024-06-03T05:16:24.635 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:16:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:16:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:16:33.925 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:16:34.104 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-radosgw-debuginfo-19.0.0-2946.g3d89accf.el 20 MB/s | 187 MB 00:09 2024-06-03T05:16:34.104 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:16:34.105 INFO:teuthology.orchestra.run.smithi064.stdout:Total 20 MB/s | 187 MB 00:09 2024-06-03T05:16:34.105 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:16:34.109 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:16:34.109 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:16:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:16:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:16:34.169 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:16:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:16:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:16:34.170 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:16:34.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-06-03T05:16:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:03:49 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:16:35.135 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:16:35.136 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2024-06-03T05:16:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:16:35.136 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2024-06-03T05:16:35.136 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:16:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-03T05:16:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:16:35.137 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:16:35.137 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2024-06-03T05:16:35.137 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:16:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:16:35.137 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 1.5 M 2024-06-03T05:16:35.137 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 6.6 M 2024-06-03T05:16:35.138 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:16:35.295 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:16:35.295 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:16:35.615 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-29 3.1 MB/s | 1.5 MB 00:00 2024-06-03T05:16:35.615 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:16:35.616 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.0 MB/s | 1.5 MB 00:00 2024-06-03T05:16:35.616 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:16:35.619 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:16:35.620 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:16:35.666 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:16:35.666 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:16:36.052 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:16:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2946. 1/1 2024-06-03T05:16:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-2946. 1/1 2024-06-03T05:16:36.878 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2946. 1/1 2024-06-03T05:16:36.879 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:16:36.879 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:16:36.879 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:16:36.879 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:16:36.879 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:16:37.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw-debuginfo 2024-06-03T05:16:37.633 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:03:51 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:16:37.762 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:16:37.763 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:16:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:16:37.763 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:16:37.763 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:16:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 187 M 2024-06-03T05:16:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:16:37.764 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:16:37.764 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:16:37.764 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:16:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:16:37.764 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 187 M 2024-06-03T05:16:37.764 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 797 M 2024-06-03T05:16:37.764 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:16:46.467 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-radosgw-debuginfo-19.0.0-2946.g3d89accf.el 22 MB/s | 187 MB 00:08 2024-06-03T05:16:46.468 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:16:46.468 INFO:teuthology.orchestra.run.smithi038.stdout:Total 22 MB/s | 187 MB 00:08 2024-06-03T05:16:46.469 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:16:46.473 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:16:46.473 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:16:47.659 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:16:47.659 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:16:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:16:55.719 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 1/1 2024-06-03T05:17:00.459 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 1/1 2024-06-03T05:17:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 1/1 2024-06-03T05:17:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:17:00.684 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:17:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:17:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:17:00.684 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:17:00.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test-debuginfo 2024-06-03T05:17:01.463 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:16 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:17:01.591 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:17:01.592 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:17:01.592 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:17:01.592 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:17:01.592 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:17:01.592 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 692 M 2024-06-03T05:17:01.593 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:17:01.593 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:17:01.593 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:17:01.593 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:17:01.593 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:17:01.593 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 692 M 2024-06-03T05:17:01.593 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 3.2 G 2024-06-03T05:17:01.593 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:17:07.953 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:17:08.049 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 1/1 2024-06-03T05:17:13.041 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 1/1 2024-06-03T05:17:13.273 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 1/1 2024-06-03T05:17:13.273 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:17:13.273 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:17:13.273 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:17:13.273 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:17:13.273 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:17:13.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test-debuginfo 2024-06-03T05:17:14.059 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:04:28 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:17:14.188 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:17:14.189 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:17:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:17:14.189 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:17:14.189 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:17:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 692 M 2024-06-03T05:17:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:17:14.190 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:17:14.190 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:17:14.190 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:17:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:17:14.190 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 692 M 2024-06-03T05:17:14.190 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 3.2 G 2024-06-03T05:17:14.190 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:17:38.968 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-test-debuginfo-19.0.0-2946.g3d89accf.el9.x 19 MB/s | 692 MB 00:37 2024-06-03T05:17:38.969 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:17:38.969 INFO:teuthology.orchestra.run.smithi064.stdout:Total 19 MB/s | 692 MB 00:37 2024-06-03T05:17:38.969 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:17:38.975 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:17:38.975 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:17:43.256 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:17:43.257 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:17:53.965 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-test-debuginfo-19.0.0-2946.g3d89accf.el9.x 17 MB/s | 692 MB 00:39 2024-06-03T05:17:53.965 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:17:53.966 INFO:teuthology.orchestra.run.smithi038.stdout:Total 17 MB/s | 692 MB 00:39 2024-06-03T05:17:53.966 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:17:53.971 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:17:53.971 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:17:58.254 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:17:58.254 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:19:00.485 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:19:00.595 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:19:08.472 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:19:08.725 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:19:08.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:08.725 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:19:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:19:08.726 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:08.726 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:19:08.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base-debuginfo 2024-06-03T05:19:09.594 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:06:24 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:19:09.680 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:19:09.724 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:19:09.725 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-06-03T05:19:09.726 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:19:09.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-debuginfo 2024-06-03T05:19:10.400 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:06:25 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:19:10.528 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:19:10.529 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:10.529 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:19:10.529 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:10.529 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:19:10.529 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 20 M 2024-06-03T05:19:10.529 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:10.530 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:19:10.530 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:10.530 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:19:10.530 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:10.530 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 20 M 2024-06-03T05:19:10.530 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 104 M 2024-06-03T05:19:10.530 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:19:11.813 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-mgr-debuginfo-19.0.0-2946.g3d89accf.el9.x8 16 MB/s | 20 MB 00:01 2024-06-03T05:19:11.814 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:19:11.814 INFO:teuthology.orchestra.run.smithi064.stdout:Total 16 MB/s | 20 MB 00:01 2024-06-03T05:19:11.814 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:19:11.818 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:19:11.818 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:19:11.978 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:19:11.978 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:19:14.386 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:19:14.506 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:15.423 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:19:15.535 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:19:15.590 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:15.832 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:19:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:19:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:15.832 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:19:15.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mds-debuginfo 2024-06-03T05:19:16.595 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:06:31 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:19:16.725 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:19:16.725 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 53 M 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 53 M 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 195 M 2024-06-03T05:19:16.726 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:19:19.319 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-mds-debuginfo-19.0.0-2946.g3d89accf.el9.x8 21 MB/s | 53 MB 00:02 2024-06-03T05:19:19.319 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:19:19.320 INFO:teuthology.orchestra.run.smithi064.stdout:Total 21 MB/s | 53 MB 00:02 2024-06-03T05:19:19.320 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:19:19.324 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:19:19.324 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:19:19.690 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:19:19.690 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:19:24.917 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:19:25.175 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:19:25.175 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:25.175 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:19:25.175 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:19:25.175 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:25.175 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:19:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:19:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:25.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base-debuginfo 2024-06-03T05:19:26.063 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:06:40 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:19:26.147 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 is already installed. 2024-06-03T05:19:26.191 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:19:26.192 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-06-03T05:19:26.192 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:19:26.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-debuginfo 2024-06-03T05:19:26.850 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:26.869 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:06:40 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:19:26.998 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:19:26.999 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:19:26.999 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:19:26.999 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:19:26.999 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:19:27.000 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 20 M 2024-06-03T05:19:27.000 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:27.000 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:19:27.000 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:19:27.000 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:19:27.000 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:27.000 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 20 M 2024-06-03T05:19:27.000 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 104 M 2024-06-03T05:19:27.000 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:19:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:27.090 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:27.091 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:19:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:19:27.091 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:27.091 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:19:27.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mon-debuginfo 2024-06-03T05:19:27.858 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:06:42 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:19:27.988 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 M 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:27.989 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 99 M 2024-06-03T05:19:27.990 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 450 M 2024-06-03T05:19:27.990 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:19:28.111 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-mgr-debuginfo-19.0.0-2946.g3d89accf.el9.x8 18 MB/s | 20 MB 00:01 2024-06-03T05:19:28.112 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:19:28.113 INFO:teuthology.orchestra.run.smithi038.stdout:Total 18 MB/s | 20 MB 00:01 2024-06-03T05:19:28.113 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:19:28.117 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:19:28.117 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:19:28.280 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:19:28.280 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:19:30.678 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:19:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:31.771 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:32.012 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:32.012 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:32.013 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:19:32.013 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:19:32.013 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:32.013 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:19:32.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mds-debuginfo 2024-06-03T05:19:32.789 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:06:46 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:19:32.919 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:19:32.919 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:19:32.919 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:19:32.919 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:19:32.919 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:19:32.920 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 53 M 2024-06-03T05:19:32.920 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:32.920 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:19:32.920 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:19:32.920 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:19:32.920 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:32.920 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 53 M 2024-06-03T05:19:32.921 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 195 M 2024-06-03T05:19:32.921 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:19:34.129 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-mon-debuginfo-19.0.0-2946.g3d89accf.el9.x8 16 MB/s | 99 MB 00:06 2024-06-03T05:19:34.130 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:19:34.130 INFO:teuthology.orchestra.run.smithi064.stdout:Total 16 MB/s | 99 MB 00:06 2024-06-03T05:19:34.131 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:19:34.135 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:19:34.135 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:19:34.779 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:19:34.779 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:19:35.798 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-mds-debuginfo-19.0.0-2946.g3d89accf.el9.x8 19 MB/s | 53 MB 00:02 2024-06-03T05:19:35.802 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:19:35.802 INFO:teuthology.orchestra.run.smithi038.stdout:Total 19 MB/s | 53 MB 00:02 2024-06-03T05:19:35.802 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:19:35.803 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:19:35.803 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:19:36.168 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:19:36.169 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:19:41.776 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:19:41.873 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:43.356 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:43.653 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:43.654 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:43.654 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:19:43.654 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:19:43.654 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:43.654 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:19:43.814 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mon-debuginfo 2024-06-03T05:19:44.417 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:06:58 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:19:44.546 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:19:44.546 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 M 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 99 M 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 450 M 2024-06-03T05:19:44.547 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:19:45.725 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:19:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:48.710 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:48.960 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:19:48.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:48.960 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:19:48.960 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:19:48.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:48.961 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:19:49.091 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-mon-debuginfo-19.0.0-2946.g3d89accf.el9.x8 22 MB/s | 99 MB 00:04 2024-06-03T05:19:49.092 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:19:49.093 INFO:teuthology.orchestra.run.smithi038.stdout:Total 22 MB/s | 99 MB 00:04 2024-06-03T05:19:49.093 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:19:49.097 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:19:49.097 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:19:49.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-osd-debuginfo 2024-06-03T05:19:49.727 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:07:04 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:19:49.742 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:19:49.742 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:19:49.856 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 288 M 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:19:49.857 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 288 M 2024-06-03T05:19:49.858 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.2 G 2024-06-03T05:19:49.858 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:20:00.654 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:20:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:20:03.047 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-osd-debuginfo-19.0.0-2946.g3d89accf.el9.x8 22 MB/s | 288 MB 00:13 2024-06-03T05:20:03.047 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:20:03.048 INFO:teuthology.orchestra.run.smithi064.stdout:Total 22 MB/s | 288 MB 00:13 2024-06-03T05:20:03.048 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:20:03.052 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:20:03.053 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:20:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:20:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:20:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:20:04.182 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:20:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:20:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:20:04.182 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:20:04.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-osd-debuginfo 2024-06-03T05:20:04.856 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:20:04.857 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:20:05.128 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:07:19 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:20:05.260 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:20:05.261 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:20:05.261 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:20:05.261 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:20:05.261 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:20:05.261 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 288 M 2024-06-03T05:20:05.261 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:20:05.261 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:20:05.261 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:20:05.262 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:20:05.262 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:20:05.262 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 288 M 2024-06-03T05:20:05.262 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.2 G 2024-06-03T05:20:05.262 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:20:28.859 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-osd-debuginfo-19.0.0-2946.g3d89accf.el9.x8 12 MB/s | 288 MB 00:23 2024-06-03T05:20:28.859 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:20:28.860 INFO:teuthology.orchestra.run.smithi038.stdout:Total 12 MB/s | 288 MB 00:23 2024-06-03T05:20:28.860 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:20:28.864 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:20:28.864 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:20:30.670 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:20:30.671 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:20:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:20:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:20:43.523 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:20:43.763 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:20:43.763 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:43.763 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:20:43.763 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:20:43.763 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:43.764 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:20:44.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse-debuginfo 2024-06-03T05:20:44.745 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:07:59 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:20:44.873 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:20:44.874 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:20:44.874 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:44.874 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:20:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.0 M 2024-06-03T05:20:44.874 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:44.874 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:20:44.874 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:44.875 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:20:44.875 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:44.875 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.0 M 2024-06-03T05:20:44.875 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 29 M 2024-06-03T05:20:44.875 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:20:45.397 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-fuse-debuginfo-19.0.0-2946.g3d89accf.el9.x 12 MB/s | 6.0 MB 00:00 2024-06-03T05:20:45.397 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:20:45.398 INFO:teuthology.orchestra.run.smithi064.stdout:Total 11 MB/s | 6.0 MB 00:00 2024-06-03T05:20:45.398 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:20:45.402 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:20:45.402 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:20:45.476 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:20:45.476 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:20:46.254 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:20:46.364 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:20:47.028 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:20:47.261 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:20:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:47.262 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:20:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:20:47.262 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:47.262 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:20:47.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel-debuginfo 2024-06-03T05:20:48.027 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:03 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:20:48.157 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 109 k 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:48.158 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 109 k 2024-06-03T05:20:48.159 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 440 k 2024-06-03T05:20:48.159 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:20:48.374 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-debuginfo-19.0.0-2946.g3d89accf. 503 kB/s | 109 kB 00:00 2024-06-03T05:20:48.375 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:20:48.375 INFO:teuthology.orchestra.run.smithi064.stdout:Total 500 kB/s | 109 kB 00:00 2024-06-03T05:20:48.375 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:20:48.379 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:20:48.379 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:20:48.416 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:20:48.417 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:20:48.540 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:20:48.634 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:20:48.987 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:20:49.230 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:20:49.230 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:49.230 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:20:49.230 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:20:49.230 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:49.230 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:20:49.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2-debuginfo 2024-06-03T05:20:50.016 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:05 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:20:50.148 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:20:50.149 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:20:50.149 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:50.149 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:20:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-03T05:20:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:50.150 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:20:50.150 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:50.150 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:20:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:50.151 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 5.4 M 2024-06-03T05:20:50.151 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 25 M 2024-06-03T05:20:50.151 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:20:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs2-debuginfo-19.0.0-2946.g3d89accf.el9. 11 MB/s | 5.4 MB 00:00 2024-06-03T05:20:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:20:50.654 INFO:teuthology.orchestra.run.smithi064.stdout:Total 11 MB/s | 5.4 MB 00:00 2024-06-03T05:20:50.654 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:20:50.658 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:20:50.658 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:20:50.728 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:20:50.728 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:20:51.440 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:20:51.553 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:20:52.069 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:20:52.303 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:20:52.303 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:52.303 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:20:52.303 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:20:52.304 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:52.304 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:20:52.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2-debuginfo 2024-06-03T05:20:53.062 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:08 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:20:53.191 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:20:53.191 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 M 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 45 M 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 256 M 2024-06-03T05:20:53.192 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:20:55.698 INFO:teuthology.orchestra.run.smithi064.stdout:librados2-debuginfo-19.0.0-2946.g3d89accf.el9.x 18 MB/s | 45 MB 00:02 2024-06-03T05:20:55.699 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:20:55.699 INFO:teuthology.orchestra.run.smithi064.stdout:Total 18 MB/s | 45 MB 00:02 2024-06-03T05:20:55.699 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:20:55.704 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:20:55.704 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:20:56.024 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:20:56.024 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:21:01.471 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:21:01.574 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:21:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:02.106 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:21:03.342 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:21:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:21:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:03.576 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:21:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:03.576 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:03.576 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:21:03.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1-debuginfo 2024-06-03T05:21:04.373 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:19 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:21:04.501 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 52 M 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:21:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:04.503 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 52 M 2024-06-03T05:21:04.503 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 308 M 2024-06-03T05:21:04.503 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:21:07.017 INFO:teuthology.orchestra.run.smithi064.stdout:librbd1-debuginfo-19.0.0-2946.g3d89accf.el9.x86 21 MB/s | 52 MB 00:02 2024-06-03T05:21:07.018 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:07.018 INFO:teuthology.orchestra.run.smithi064.stdout:Total 21 MB/s | 52 MB 00:02 2024-06-03T05:21:07.018 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:21:07.023 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:21:07.023 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:21:07.385 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:21:07.386 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:21:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:21:10.078 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:21:10.078 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:10.078 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:10.078 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:10.078 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:10.078 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:10.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse-debuginfo 2024-06-03T05:21:10.889 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:08:24 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:11.019 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:11.020 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:11.020 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:11.020 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:11.020 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:11.020 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.0 M 2024-06-03T05:21:11.020 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:11.020 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:11.020 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:11.020 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:11.021 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:11.021 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 6.0 M 2024-06-03T05:21:11.021 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 29 M 2024-06-03T05:21:11.021 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:11.525 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-fuse-debuginfo-19.0.0-2946.g3d89accf.el9.x 12 MB/s | 6.0 MB 00:00 2024-06-03T05:21:11.526 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:11.526 INFO:teuthology.orchestra.run.smithi038.stdout:Total 12 MB/s | 6.0 MB 00:00 2024-06-03T05:21:11.526 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:11.530 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:11.530 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:11.604 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:11.604 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:12.485 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:21:13.016 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:21:13.258 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:21:13.258 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:13.258 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:13.258 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:13.258 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:13.258 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:13.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel-debuginfo 2024-06-03T05:21:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:21:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:21:14.038 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:08:28 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:14.168 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:14.168 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:14.169 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:14.169 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 109 k 2024-06-03T05:21:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:14.169 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:14.169 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:14.169 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:14.170 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 109 k 2024-06-03T05:21:14.170 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 440 k 2024-06-03T05:21:14.170 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:14.403 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-debuginfo-19.0.0-2946.g3d89accf. 467 kB/s | 109 kB 00:00 2024-06-03T05:21:14.403 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:14.404 INFO:teuthology.orchestra.run.smithi038.stdout:Total 464 kB/s | 109 kB 00:00 2024-06-03T05:21:14.404 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:14.407 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:14.407 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:14.446 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:14.447 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:14.567 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:14.670 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:21:15.040 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:21:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:21:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:15.283 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:15.283 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:15.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2-debuginfo 2024-06-03T05:21:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:21:16.043 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:08:30 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:16.173 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:16.173 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:16.173 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:16.173 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 5.4 M 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 25 M 2024-06-03T05:21:16.174 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:21:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:16.268 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:21:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:16.269 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:21:16.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs-debuginfo 2024-06-03T05:21:16.831 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs2-debuginfo-19.0.0-2946.g3d89accf.el9. 8.2 MB/s | 5.4 MB 00:00 2024-06-03T05:21:16.831 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:16.832 INFO:teuthology.orchestra.run.smithi038.stdout:Total 8.2 MB/s | 5.4 MB 00:00 2024-06-03T05:21:16.832 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:16.836 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:16.836 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:16.906 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:16.906 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:17.044 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:32 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:21:17.174 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:21:17.175 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:17.175 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:21:17.175 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:17.175 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:21:17.175 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 615 k 2024-06-03T05:21:17.175 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:17.176 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:21:17.176 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:17.176 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:21:17.176 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:17.176 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 615 k 2024-06-03T05:21:17.176 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.1 M 2024-06-03T05:21:17.176 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:21:17.462 INFO:teuthology.orchestra.run.smithi064.stdout:python3-cephfs-debuginfo-19.0.0-2946.g3d89accf. 2.1 MB/s | 615 kB 00:00 2024-06-03T05:21:17.462 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:17.463 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.1 MB/s | 615 kB 00:00 2024-06-03T05:21:17.463 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:21:17.466 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:21:17.466 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:21:17.508 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:21:17.509 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:21:17.609 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:17.682 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:21:17.715 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:21:17.780 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:21:18.199 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:21:18.298 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:21:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:21:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:18.449 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:18.449 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:21:18.525 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:18.525 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:21:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:18.526 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:18.526 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:21:18.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2-debuginfo 2024-06-03T05:21:18.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados-debuginfo 2024-06-03T05:21:19.237 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:08:33 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:19.308 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:34 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:21:19.367 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 M 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 45 M 2024-06-03T05:21:19.368 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 256 M 2024-06-03T05:21:19.369 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 771 k 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:19.440 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:21:19.441 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:19.441 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 771 k 2024-06-03T05:21:19.441 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.7 M 2024-06-03T05:21:19.441 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:21:19.965 INFO:teuthology.orchestra.run.smithi064.stdout:python3-rados-debuginfo-19.0.0-2946.g3d89accf.e 1.4 MB/s | 771 kB 00:00 2024-06-03T05:21:19.966 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:19.966 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.4 MB/s | 771 kB 00:00 2024-06-03T05:21:19.966 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:21:19.969 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:21:19.969 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:21:20.012 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:21:20.012 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:21:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:21:20.316 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9. 1/1 2024-06-03T05:21:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9. 1/1 2024-06-03T05:21:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9. 1/1 2024-06-03T05:21:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:21.051 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:21:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:21.052 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:21:21.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd-debuginfo 2024-06-03T05:21:21.617 INFO:teuthology.orchestra.run.smithi038.stdout:librados2-debuginfo-19.0.0-2946.g3d89accf.el9.x 20 MB/s | 45 MB 00:02 2024-06-03T05:21:21.618 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:21.619 INFO:teuthology.orchestra.run.smithi038.stdout:Total 20 MB/s | 45 MB 00:02 2024-06-03T05:21:21.619 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:21.623 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:21.623 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:21.829 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:36 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:21:21.943 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:21.944 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:21.959 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:21:21.959 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:21.959 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:21:21.959 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:21.959 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:21:21.959 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.1 M 2024-06-03T05:21:21.959 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:21.959 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:21:21.960 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:21.960 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:21:21.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:21.960 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.1 M 2024-06-03T05:21:21.960 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 4.3 M 2024-06-03T05:21:21.960 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:21:22.401 INFO:teuthology.orchestra.run.smithi064.stdout:python3-rbd-debuginfo-19.0.0-2946.g3d89accf.el9 2.6 MB/s | 1.1 MB 00:00 2024-06-03T05:21:22.402 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:22.402 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.6 MB/s | 1.1 MB 00:00 2024-06-03T05:21:22.402 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:21:22.405 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:21:22.405 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:21:22.449 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:21:22.449 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:21:22.687 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:21:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:23.252 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:23.501 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:21:23.501 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:23.502 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:23.502 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:21:23.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw-debuginfo 2024-06-03T05:21:24.286 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:39 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:21:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:21:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:21:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:21:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 337 k 2024-06-03T05:21:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:21:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:24.415 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:21:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:24.416 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 337 k 2024-06-03T05:21:24.416 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.0 M 2024-06-03T05:21:24.416 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:21:24.679 INFO:teuthology.orchestra.run.smithi064.stdout:python3-rgw-debuginfo-19.0.0-2946.g3d89accf.el9 1.3 MB/s | 337 kB 00:00 2024-06-03T05:21:24.679 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:24.680 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.2 MB/s | 337 kB 00:00 2024-06-03T05:21:24.680 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:21:24.685 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:21:24.685 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:21:24.723 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:21:24.724 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:21:24.869 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:21:24.981 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:25.582 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:25.582 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:25.582 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:21:25.583 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:25.583 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:25.583 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:21:25.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse-debuginfo 2024-06-03T05:21:26.374 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:41 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:21:26.503 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:21:26.503 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:21:26.504 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:26.504 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:21:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 516 k 2024-06-03T05:21:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:26.504 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:21:26.504 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:26.504 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:21:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:26.505 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 516 k 2024-06-03T05:21:26.505 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.6 M 2024-06-03T05:21:26.505 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:21:27.024 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-debuginfo-19.0.0-2946.g3d89accf.el9.x8 995 kB/s | 516 kB 00:00 2024-06-03T05:21:27.024 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:27.025 INFO:teuthology.orchestra.run.smithi064.stdout:Total 992 kB/s | 516 kB 00:00 2024-06-03T05:21:27.025 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:21:27.028 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:21:27.028 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:21:27.069 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:21:27.069 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:21:27.234 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:21:27.342 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:21:27.380 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:27.485 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:21:27.663 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:21:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:21:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:27.908 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:21:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:27.908 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:27.908 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:21:28.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror-debuginfo 2024-06-03T05:21:28.665 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:08:43 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:21:28.793 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:21:28.793 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:28.794 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:21:28.794 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:28.794 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:21:28.794 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 52 M 2024-06-03T05:21:28.794 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:28.794 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:21:28.794 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:28.794 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:21:28.794 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:28.795 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 52 M 2024-06-03T05:21:28.795 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 298 M 2024-06-03T05:21:28.795 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:21:29.412 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:21:29.647 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_ 1/1 2024-06-03T05:21:29.647 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:29.648 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:29.648 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:29.648 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:29.648 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:29.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1-debuginfo 2024-06-03T05:21:30.422 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:08:44 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:30.550 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:30.551 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:30.551 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:30.552 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 52 M 2024-06-03T05:21:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:30.552 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:30.553 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:30.553 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:30.553 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 52 M 2024-06-03T05:21:30.553 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 308 M 2024-06-03T05:21:30.554 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:35.078 INFO:teuthology.orchestra.run.smithi038.stdout:librbd1-debuginfo-19.0.0-2946.g3d89accf.el9.x86 11 MB/s | 52 MB 00:04 2024-06-03T05:21:35.079 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:35.079 INFO:teuthology.orchestra.run.smithi038.stdout:Total 11 MB/s | 52 MB 00:04 2024-06-03T05:21:35.080 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:35.084 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:35.084 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:35.447 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:35.447 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:38.485 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-debuginfo-19.0.0-2946.g3d89accf.el9. 5.3 MB/s | 52 MB 00:09 2024-06-03T05:21:38.486 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:38.486 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.3 MB/s | 52 MB 00:09 2024-06-03T05:21:38.486 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:21:38.490 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:21:38.491 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:21:38.845 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:21:38.845 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:21:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:41.912 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:21:44.141 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:21:44.378 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:21:44.378 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:44.378 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:44.378 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:44.379 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:44.379 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:44.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs-debuginfo 2024-06-03T05:21:45.117 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:21:45.172 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:08:59 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:45.219 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:21:45.302 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:45.302 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 615 k 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:45.303 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 615 k 2024-06-03T05:21:45.304 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.1 M 2024-06-03T05:21:45.304 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:45.590 INFO:teuthology.orchestra.run.smithi038.stdout:python3-cephfs-debuginfo-19.0.0-2946.g3d89accf. 2.1 MB/s | 615 kB 00:00 2024-06-03T05:21:45.591 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:45.591 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.1 MB/s | 615 kB 00:00 2024-06-03T05:21:45.591 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:45.594 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:45.595 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:45.635 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:45.635 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:45.807 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:45.899 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:21:46.290 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:21:46.536 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9 1/1 2024-06-03T05:21:46.536 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:46.536 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:46.536 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:46.536 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:46.536 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:46.696 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados-debuginfo 2024-06-03T05:21:47.276 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:21:47.303 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:09:01 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:47.433 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 771 k 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 771 k 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.7 M 2024-06-03T05:21:47.434 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:47.496 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:21:47.496 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:47.496 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:21:47.496 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:47.496 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:47.497 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:21:47.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd-debuginfo 2024-06-03T05:21:47.868 INFO:teuthology.orchestra.run.smithi038.stdout:python3-rados-debuginfo-19.0.0-2946.g3d89accf.e 1.7 MB/s | 771 kB 00:00 2024-06-03T05:21:47.869 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:47.869 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.7 MB/s | 771 kB 00:00 2024-06-03T05:21:47.869 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:47.872 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:47.872 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:47.914 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:47.915 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:48.115 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:48.211 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9. 1/1 2024-06-03T05:21:48.276 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:09:03 ago on Mon 03 Jun 2024 05:12:45 AM UTC. 2024-06-03T05:21:48.403 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-06-03T05:21:48.404 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-06-03T05:21:48.404 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:48.404 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-06-03T05:21:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.3 M 2024-06-03T05:21:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:48.404 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-06-03T05:21:48.404 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-06-03T05:21:48.404 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-06-03T05:21:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:48.405 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.3 M 2024-06-03T05:21:48.405 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 5.7 M 2024-06-03T05:21:48.405 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-06-03T05:21:48.581 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9. 1/1 2024-06-03T05:21:48.712 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-debuginfo-19.0.0-2946.g3d89accf.el9.x86 4.2 MB/s | 1.3 MB 00:00 2024-06-03T05:21:48.713 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:48.713 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.2 MB/s | 1.3 MB 00:00 2024-06-03T05:21:48.713 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-06-03T05:21:48.716 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-06-03T05:21:48.717 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-06-03T05:21:48.762 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-06-03T05:21:48.763 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-06-03T05:21:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9. 1/1 2024-06-03T05:21:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:48.870 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:48.871 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:48.871 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:49.024 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-06-03T05:21:49.040 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd-debuginfo 2024-06-03T05:21:49.130 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:21:49.511 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:09:03 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:49.738 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:21:49.738 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:49.739 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-06-03T05:21:49.739 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:49.739 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:21:49.739 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-06-03T05:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.1 M 2024-06-03T05:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 1.1 M 2024-06-03T05:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 4.3 M 2024-06-03T05:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:49.932 DEBUG:teuthology.parallel:result is None 2024-06-03T05:21:50.079 INFO:teuthology.orchestra.run.smithi038.stdout:python3-rbd-debuginfo-19.0.0-2946.g3d89accf.el9 3.7 MB/s | 1.1 MB 00:00 2024-06-03T05:21:50.080 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:50.080 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.7 MB/s | 1.1 MB 00:00 2024-06-03T05:21:50.080 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:50.084 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:50.084 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:50.129 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:50.129 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:50.489 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:50.962 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:51.277 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:51.277 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:51.277 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:51.277 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:51.277 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:51.277 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:51.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw-debuginfo 2024-06-03T05:21:52.090 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:09:06 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:52.219 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:52.220 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:52.220 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:52.220 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 337 k 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 337 k 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.0 M 2024-06-03T05:21:52.221 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:52.616 INFO:teuthology.orchestra.run.smithi038.stdout:python3-rgw-debuginfo-19.0.0-2946.g3d89accf.el9 855 kB/s | 337 kB 00:00 2024-06-03T05:21:52.616 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:52.617 INFO:teuthology.orchestra.run.smithi038.stdout:Total 852 kB/s | 337 kB 00:00 2024-06-03T05:21:52.617 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:52.620 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:52.620 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:52.662 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:52.662 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:52.803 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:52.909 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:53.294 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 1/1 2024-06-03T05:21:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:53.535 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:53.535 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:53.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse-debuginfo 2024-06-03T05:21:54.324 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:09:08 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:54.452 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:54.453 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:54.453 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:54.453 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:54.453 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:54.453 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 516 k 2024-06-03T05:21:54.453 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:54.454 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:54.454 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:54.454 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:54.454 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:54.454 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 516 k 2024-06-03T05:21:54.454 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.6 M 2024-06-03T05:21:54.454 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:21:54.852 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-debuginfo-19.0.0-2946.g3d89accf.el9.x8 1.3 MB/s | 516 kB 00:00 2024-06-03T05:21:54.853 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:21:54.853 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.3 MB/s | 516 kB 00:00 2024-06-03T05:21:54.853 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:21:54.857 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:21:54.857 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:21:54.898 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:21:54.898 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:21:55.068 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:21:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:21:55.530 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:21:55.801 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 1/1 2024-06-03T05:21:55.801 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:55.801 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:21:55.801 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:21:55.801 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:55.801 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:21:55.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror-debuginfo 2024-06-03T05:21:56.570 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:09:10 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:21:56.700 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 52 M 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 52 M 2024-06-03T05:21:56.701 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 298 M 2024-06-03T05:21:56.702 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:22:01.039 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-debuginfo-19.0.0-2946.g3d89accf.el9. 12 MB/s | 52 MB 00:04 2024-06-03T05:22:01.039 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:22:01.040 INFO:teuthology.orchestra.run.smithi038.stdout:Total 12 MB/s | 52 MB 00:04 2024-06-03T05:22:01.040 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:22:01.044 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:22:01.044 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:22:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:22:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:22:07.667 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:22:07.772 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:22:09.861 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:22:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86 1/1 2024-06-03T05:22:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:10.108 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:22:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:22:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:10.109 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:22:10.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd-debuginfo 2024-06-03T05:22:10.875 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:09:24 ago on Mon 03 Jun 2024 05:12:46 AM UTC. 2024-06-03T05:22:11.006 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-06-03T05:22:11.007 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:22:11.007 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-06-03T05:22:11.007 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:22:11.007 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-06-03T05:22:11.007 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.3 M 2024-06-03T05:22:11.007 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:11.007 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-06-03T05:22:11.008 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-06-03T05:22:11.008 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-06-03T05:22:11.008 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:11.008 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 1.3 M 2024-06-03T05:22:11.008 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 5.7 M 2024-06-03T05:22:11.008 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-06-03T05:22:12.366 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-debuginfo-19.0.0-2946.g3d89accf.el9.x86 968 kB/s | 1.3 MB 00:01 2024-06-03T05:22:12.366 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-06-03T05:22:12.367 INFO:teuthology.orchestra.run.smithi038.stdout:Total 967 kB/s | 1.3 MB 00:01 2024-06-03T05:22:12.367 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-06-03T05:22:12.371 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-06-03T05:22:12.371 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-06-03T05:22:12.422 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-06-03T05:22:12.423 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-06-03T05:22:12.678 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-06-03T05:22:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:22:13.154 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:22:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 1/1 2024-06-03T05:22:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:13.399 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-06-03T05:22:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-03T05:22:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:13.399 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-06-03T05:22:13.591 DEBUG:teuthology.parallel:result is None 2024-06-03T05:22:13.592 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T05:22:13.821 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-03T05:22:13.844 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-2946.g3d89accf.el9 2024-06-03T05:22:13.845 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-03T05:22:13.845 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-03T05:22:13.847 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-03T05:22:13.992 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-03T05:22:14.019 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-2946.g3d89accf.el9 2024-06-03T05:22:14.020 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-03T05:22:14.020 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-03T05:22:14.021 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-03T05:22:14.021 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:14.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-03T05:22:14.049 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:14.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-03T05:22:14.090 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-03T05:22:14.091 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:14.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-06-03T05:22:14.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-03T05:22:14.186 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:14.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-06-03T05:22:14.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-03T05:22:14.287 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-03T05:22:14.287 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:14.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-03T05:22:14.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-03T05:22:14.380 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:14.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-03T05:22:14.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-03T05:22:14.477 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-03T05:22:14.477 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:14.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-06-03T05:22:14.503 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-03T05:22:14.569 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:14.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-06-03T05:22:14.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-03T05:22:14.665 INFO:teuthology.run_tasks:Running task ceph... 2024-06-03T05:22:14.745 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-06-03T05:22:14.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /var/log/ceph 2024-06-03T05:22:14.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-06-03T05:22:14.776 INFO:tasks.ceph:Disabling ceph logrotate... 2024-06-03T05:22:14.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-03T05:22:14.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-03T05:22:14.842 INFO:tasks.ceph:Creating extra log directories... 2024-06-03T05:22:14.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-03T05:22:14.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-03T05:22:14.919 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-06-03T05:22:14.919 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-06-03T05:22:14.919 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436', 'branch': 'squid', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/valgrind}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7738436', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'dfcc4b3c2c55532c4c04fa47551c9df1dffdf746'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5132, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'sleep_before_teardown': 0, 'subset': '20949/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'dfcc4b3c2c55532c4c04fa47551c9df1dffdf746', 'targets': {'smithi038.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIwl5p2Yl/+R2k5UW83Q/3ZbmRKpKyh0MBcBIImYmNkv+QajZidIFJySEPOBL0apnqYE/VG5YuLn3bkuVHJKkvQ=', 'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAFjBxwOlDeyKlvgiJAe0zBy0004ZNp0GEi6F0ppdc4Vwd3os3jdqmrzsToXFZqcM8uZMVOhA9EbQDe7+4l3Gos='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'mon_recovery': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '861a8dcf7aa816a26e13f039336f7ed0a9aec0fa', 'timestamp': '2024-06-02_21:00:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-06-03T05:22:14.920 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-03T05:22:14.949 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-03T05:22:14.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2024-06-03T05:22:15.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-06-03T05:22:15.032 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:15.032 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-06-03T05:22:15.084 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-03T05:22:15.085 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-06-03T05:22:15.140 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-03T05:22:15.140 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T05:22:15.140 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-06-03T05:22:15.140 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T05:22:15.141 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T05:22:15.141 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-06-03 05:14:38.070426005 +0000 2024-06-03T05:22:15.141 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-06-03 05:11:48.109582803 +0000 2024-06-03T05:22:15.141 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-06-03 05:11:48.109582803 +0000 2024-06-03T05:22:15.141 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-06-03 05:11:48.109582803 +0000 2024-06-03T05:22:15.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-03T05:22:15.204 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-06-03T05:22:15.204 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-06-03T05:22:15.204 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00029319 s, 1.7 MB/s 2024-06-03T05:22:15.205 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-03T05:22:15.262 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-06-03T05:22:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-03T05:22:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T05:22:15.318 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-06-03T05:22:15.318 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T05:22:15.318 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T05:22:15.318 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-06-03 05:14:38.070426005 +0000 2024-06-03T05:22:15.318 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-06-03 05:11:48.446576436 +0000 2024-06-03T05:22:15.318 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-06-03 05:11:48.446576436 +0000 2024-06-03T05:22:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-06-03 05:11:48.446576436 +0000 2024-06-03T05:22:15.319 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-03T05:22:15.383 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-06-03T05:22:15.383 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-06-03T05:22:15.383 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000351873 s, 1.5 MB/s 2024-06-03T05:22:15.384 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-03T05:22:15.441 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-06-03T05:22:15.496 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-03T05:22:15.496 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T05:22:15.496 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-03T05:22:15.496 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T05:22:15.496 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T05:22:15.496 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-06-03 05:14:38.071425987 +0000 2024-06-03T05:22:15.496 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-06-03 05:11:48.814569482 +0000 2024-06-03T05:22:15.496 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-06-03 05:11:48.814569482 +0000 2024-06-03T05:22:15.496 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-06-03 05:11:48.814569482 +0000 2024-06-03T05:22:15.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-03T05:22:15.560 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-06-03T05:22:15.561 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-06-03T05:22:15.561 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000339714 s, 1.5 MB/s 2024-06-03T05:22:15.562 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-03T05:22:15.619 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-06-03T05:22:15.675 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-03T05:22:15.675 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T05:22:15.675 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-03T05:22:15.675 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T05:22:15.675 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T05:22:15.676 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-06-03 05:14:38.071425987 +0000 2024-06-03T05:22:15.676 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-06-03 05:11:49.275560772 +0000 2024-06-03T05:22:15.676 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-06-03 05:11:49.275560772 +0000 2024-06-03T05:22:15.676 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-06-03 05:11:49.275560772 +0000 2024-06-03T05:22:15.676 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-03T05:22:15.740 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-06-03T05:22:15.741 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-06-03T05:22:15.741 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000360808 s, 1.4 MB/s 2024-06-03T05:22:15.742 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-03T05:22:15.799 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:15.799 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:15.799 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-06-03T05:22:15.819 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-03T05:22:15.819 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-06-03T05:22:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-03T05:22:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T05:22:15.879 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 673 Links: 1 2024-06-03T05:22:15.879 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T05:22:15.879 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T05:22:15.879 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-06-03 05:14:34.610841910 +0000 2024-06-03T05:22:15.879 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-06-03 05:11:48.098855283 +0000 2024-06-03T05:22:15.879 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-06-03 05:11:48.098855283 +0000 2024-06-03T05:22:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-06-03 05:11:48.098855283 +0000 2024-06-03T05:22:15.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-03T05:22:15.947 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-06-03T05:22:15.947 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-06-03T05:22:15.948 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000287954 s, 1.8 MB/s 2024-06-03T05:22:15.949 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-03T05:22:16.009 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-06-03T05:22:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-03T05:22:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T05:22:16.068 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 693 Links: 1 2024-06-03T05:22:16.068 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T05:22:16.068 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T05:22:16.068 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-06-03 05:14:34.611841891 +0000 2024-06-03T05:22:16.068 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-06-03 05:11:48.443848431 +0000 2024-06-03T05:22:16.068 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-06-03 05:11:48.443848431 +0000 2024-06-03T05:22:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-06-03 05:11:48.443848431 +0000 2024-06-03T05:22:16.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-03T05:22:16.135 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-06-03T05:22:16.135 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-06-03T05:22:16.135 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000212377 s, 2.4 MB/s 2024-06-03T05:22:16.136 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-03T05:22:16.195 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-06-03T05:22:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-03T05:22:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T05:22:16.253 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 714 Links: 1 2024-06-03T05:22:16.253 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T05:22:16.253 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T05:22:16.253 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-06-03 05:14:34.611841891 +0000 2024-06-03T05:22:16.253 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-06-03 05:11:48.783841679 +0000 2024-06-03T05:22:16.254 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-06-03 05:11:48.783841679 +0000 2024-06-03T05:22:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-06-03 05:11:48.783841679 +0000 2024-06-03T05:22:16.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-03T05:22:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-06-03T05:22:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-06-03T05:22:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000192096 s, 2.7 MB/s 2024-06-03T05:22:16.322 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-03T05:22:16.380 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-06-03T05:22:16.438 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-03T05:22:16.438 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T05:22:16.438 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 732 Links: 1 2024-06-03T05:22:16.438 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T05:22:16.438 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T05:22:16.438 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-06-03 05:14:34.611841891 +0000 2024-06-03T05:22:16.439 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-06-03 05:11:49.133834726 +0000 2024-06-03T05:22:16.439 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-06-03 05:11:49.133834726 +0000 2024-06-03T05:22:16.439 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-06-03 05:11:49.133834726 +0000 2024-06-03T05:22:16.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-03T05:22:16.504 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-06-03T05:22:16.504 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-06-03T05:22:16.504 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000242088 s, 2.1 MB/s 2024-06-03T05:22:16.505 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-03T05:22:16.563 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:16.564 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-03T05:22:16.564 INFO:tasks.ceph:Generating config... 2024-06-03T05:22:16.565 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-06-03T05:22:16.565 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-06-03T05:22:16.565 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-06-03T05:22:16.566 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-06-03T05:22:16.566 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-06-03T05:22:16.566 INFO:tasks.ceph:[global] ms type = async 2024-06-03T05:22:16.566 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-06-03T05:22:16.566 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-06-03T05:22:16.566 INFO:tasks.ceph:[mgr] debug ms = 1 2024-06-03T05:22:16.566 INFO:tasks.ceph:[mon] debug mon = 20 2024-06-03T05:22:16.566 INFO:tasks.ceph:[mon] debug ms = 1 2024-06-03T05:22:16.566 INFO:tasks.ceph:[mon] debug paxos = 20 2024-06-03T05:22:16.566 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-06-03T05:22:16.567 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-06-03T05:22:16.567 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-06-03T05:22:16.567 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-06-03T05:22:16.567 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-06-03T05:22:16.567 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-06-03T05:22:16.567 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-06-03T05:22:16.567 INFO:tasks.ceph:[osd] bdev async discard = True 2024-06-03T05:22:16.567 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-06-03T05:22:16.567 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-06-03T05:22:16.567 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-06-03T05:22:16.567 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] debug monc = 20 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] debug ms = 1 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] debug osd = 20 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-06-03T05:22:16.568 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-06-03T05:22:16.569 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-06-03T05:22:16.570 INFO:tasks.ceph:Setting up mon.a... 2024-06-03T05:22:16.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-06-03T05:22:16.623 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.keyring 2024-06-03T05:22:16.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-06-03T05:22:16.673 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-03T05:22:16.699 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.38'), ('mon.c', '[v2:172.21.15.38:3301,v1:172.21.15.38:6790]'), ('mon.b', '172.21.15.64')] 2024-06-03T05:22:16.699 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.64', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-06-03T05:22:16.700 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:16.700 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-06-03T05:22:16.755 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.38 --addv c '[v2:172.21.15.38:3301,v1:172.21.15.38:6790]' --add b 172.21.15.64 --print /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:16.843 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:16.843 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: generated fsid ebc09f7e-f094-4421-826f-479fc526fc62 2024-06-03T05:22:16.843 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = quincy 2024-06-03T05:22:16.843 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 0 2024-06-03T05:22:16.843 INFO:teuthology.orchestra.run.smithi038.stdout:fsid ebc09f7e-f094-4421-826f-479fc526fc62 2024-06-03T05:22:16.844 INFO:teuthology.orchestra.run.smithi038.stdout:last_changed 2024-06-03T05:22:16.844560+0000 2024-06-03T05:22:16.844 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-06-03T05:22:16.844560+0000 2024-06-03T05:22:16.844 INFO:teuthology.orchestra.run.smithi038.stdout:min_mon_release 17 (quincy) 2024-06-03T05:22:16.844 INFO:teuthology.orchestra.run.smithi038.stdout:election_strategy: 1 2024-06-03T05:22:16.844 INFO:teuthology.orchestra.run.smithi038.stdout:0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-06-03T05:22:16.844 INFO:teuthology.orchestra.run.smithi038.stdout:1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.b 2024-06-03T05:22:16.844 INFO:teuthology.orchestra.run.smithi038.stdout:2: [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] mon.c 2024-06-03T05:22:16.844 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-06-03T05:22:16.845 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-06-03T05:22:16.861 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ebc09f7e-f094-4421-826f-479fc526fc62... 2024-06-03T05:22:16.862 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-03T05:22:16.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-03T05:22:16.945 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = cephx 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2024-06-03T05:22:16.946 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2024-06-03T05:22:16.947 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2024-06-03T05:22:16.948 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: mon health detail to clog = false 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = "172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.64" 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: mon client directed command retry = 5 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: mon election default strategy = 1 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: ms bind msgr1 = False 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: ms bind msgr2 = True 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: ms inject socket failures = 5000 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: ms type = async 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: osd heartbeat grace = 80 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = ebc09f7e-f094-4421-826f-479fc526fc62 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2024-06-03T05:22:16.949 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 600 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd mclock profile = high_recovery_ops 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug trim objects = true 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2024-06-03T05:22:16.950 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2024-06-03T05:22:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: bdev async discard = True 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: bdev enable discard = True 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore allocator = bitmap 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore block size = 96636764160 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore fsck on mount = True 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore zero block detection = True 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluefs = 1 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluestore = 1 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: debug monc = 20 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 20 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: debug rocksdb = 10 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd backfillfull_ratio = 0.85 2024-06-03T05:22:16.952 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd full ratio = 0.9 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd nearfull ratio = 0.8 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd blocked scrub grace period = 3600 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify cached snaps = True 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify missing on start = True 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd failsafe full ratio = 0.95 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd fast shutdown = False 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd max markdown count = 1000 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd mclock override recovery settings = True 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd objectstore = bluestore 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd op queue = debug_random 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd op queue cut off = debug_random 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: osd shutdown pgref assert = True 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.953 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-06-03T05:22:16.954 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 20 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.955 INFO:teuthology.orchestra.run.smithi038.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: auth mon ticket ttl = 660 # 11m 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: auth service ticket ttl = 240 # 4m 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: # don't complain about insecure global_id in the test suite 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: # 1m isn't quite enough 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: mon_down_mkfs_grace = 2m 2024-06-03T05:22:16.956 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_filestore_osds = false 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: mon min osdmap epochs = 50 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd crush smoke test = False 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: mon osdmap full prune interval = 2 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: mon osdmap full prune min = 15 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: mon osdmap full prune txsize = 2 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: mon scrub interval = 300 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: paxos service trim min = 10 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-03T05:22:16.957 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-03T05:22:16.958 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2024-06-03T05:22:16.958 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.c] 2024-06-03T05:22:16.958 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.b] 2024-06-03T05:22:16.958 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-06-03T05:22:16.958 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-06-03T05:22:16.959 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-06-03T05:22:16.960 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-06-03T05:22:16.961 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.64" 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 1 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: ms bind msgr1 = False 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: ms bind msgr2 = True 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: ms type = async 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: osd heartbeat grace = 80 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = ebc09f7e-f094-4421-826f-479fc526fc62 2024-06-03T05:22:16.962 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-06-03T05:22:16.963 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-06-03T05:22:16.964 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: bdev async discard = True 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: bdev enable discard = True 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore allocator = bitmap 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 1 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 1 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-06-03T05:22:16.965 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd blocked scrub grace period = 3600 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd fast shutdown = False 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd max markdown count = 1000 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.966 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-06-03T05:22:16.967 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.968 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: mon min osdmap epochs = 50 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd crush smoke test = False 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune interval = 2 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune min = 15 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune txsize = 2 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: paxos service trim min = 10 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-06-03T05:22:16.969 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-06-03T05:22:16.970 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-06-03T05:22:16.970 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-03T05:22:16.970 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-03T05:22:16.970 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-06-03T05:22:16.970 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-06-03T05:22:16.970 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-06-03T05:22:16.971 INFO:tasks.ceph:Creating admin key on mon.a... 2024-06-03T05:22:16.971 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-06-03T05:22:17.067 INFO:tasks.ceph:Copying monmap to all nodes... 2024-06-03T05:22:17.067 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:17.067 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-06-03T05:22:17.083 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:17.083 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-06-03T05:22:17.139 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:22:17.139 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:17.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-03T05:22:17.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-03T05:22:17.215 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:17.215 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:17.270 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:22:17.271 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:17.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-03T05:22:17.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-03T05:22:17.312 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:17.312 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:17.371 INFO:tasks.ceph:Setting up mon nodes... 2024-06-03T05:22:17.371 INFO:tasks.ceph:Setting up mgr nodes... 2024-06-03T05:22:17.371 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-06-03T05:22:17.423 INFO:teuthology.orchestra.run.smithi038.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-06-03T05:22:17.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-06-03T05:22:17.493 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-06-03T05:22:17.497 INFO:tasks.ceph:Setting up mds nodes... 2024-06-03T05:22:17.497 INFO:tasks.ceph_client:Setting up client nodes... 2024-06-03T05:22:17.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-03T05:22:17.540 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-06-03T05:22:17.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-03T05:22:17.599 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-06-03T05:22:17.613 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-06-03T05:22:17.614 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-03T05:22:17.614 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-06-03T05:22:17.639 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:17.640 INFO:tasks.ceph:role: osd.0 2024-06-03T05:22:17.640 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:22:17.640 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-03T05:22:17.732 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T05:22:17.732 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T05:22:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T05:22:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T05:22:17.733 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T05:22:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-06-03T05:22:17.733 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T05:22:17.733 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T05:22:17.733 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T05:22:17.733 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T05:22:19.077 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-06-03T05:22:19.079 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-06-03T05:22:19.079 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-06-03T05:22:19.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-06-03T05:22:19.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-06-03T05:22:19.261 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:19.261 INFO:tasks.ceph:role: osd.1 2024-06-03T05:22:19.261 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:22:19.261 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-03T05:22:19.357 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T05:22:19.357 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T05:22:19.357 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T05:22:19.357 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T05:22:19.357 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T05:22:19.357 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-06-03T05:22:19.358 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T05:22:19.358 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T05:22:19.358 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T05:22:19.358 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T05:22:20.669 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-06-03T05:22:20.671 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-06-03T05:22:20.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-06-03T05:22:20.704 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-06-03T05:22:20.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-06-03T05:22:20.842 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:20.842 INFO:tasks.ceph:role: osd.2 2024-06-03T05:22:20.842 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:22:20.843 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-03T05:22:20.935 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T05:22:20.935 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T05:22:20.935 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T05:22:20.936 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T05:22:20.936 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T05:22:20.936 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-06-03T05:22:20.936 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T05:22:20.936 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T05:22:20.936 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T05:22:20.936 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T05:22:22.253 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-06-03T05:22:22.255 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-06-03T05:22:22.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-06-03T05:22:22.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-06-03T05:22:22.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-06-03T05:22:22.438 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:22.438 INFO:tasks.ceph:role: osd.3 2024-06-03T05:22:22.438 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:22:22.438 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T05:22:22.534 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T05:22:23.879 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-06-03T05:22:23.881 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-06-03T05:22:23.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-06-03T05:22:23.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-06-03T05:22:23.995 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:24.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:24.099+0000 7f755ce13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-06-03T05:22:24.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:24.104+0000 7f755ce13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-06-03T05:22:24.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:24.104+0000 7f755ce13640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-03T05:22:24.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:24.104+0000 7f755ce13640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-03T05:22:24.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:24.104+0000 7f755ce13640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-03T05:22:24.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:24.105+0000 7f755ce13640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-06-03T05:22:26.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-03T05:22:27.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:27.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:27.094+0000 7fe8f800e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-06-03T05:22:27.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:27.094+0000 7fe8f800e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-06-03T05:22:27.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:27.094+0000 7fe8f800e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-03T05:22:27.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:27.094+0000 7fe8f800e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-03T05:22:27.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:27.094+0000 7fe8f800e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-03T05:22:27.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:27.095+0000 7fe8f800e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-06-03T05:22:29.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-03T05:22:29.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:30.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:30.082+0000 7f4c74e0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-06-03T05:22:30.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:30.082+0000 7f4c74e0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-06-03T05:22:30.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:30.082+0000 7f4c74e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-03T05:22:30.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:30.082+0000 7f4c74e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-03T05:22:30.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:30.082+0000 7f4c74e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-03T05:22:30.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:30.083+0000 7f4c74e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-06-03T05:22:32.947 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-03T05:22:32.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:33.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:33.071+0000 7f6bc2005640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-06-03T05:22:33.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:33.072+0000 7f6bc2005640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-06-03T05:22:33.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:33.072+0000 7f6bc2005640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-03T05:22:33.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:33.072+0000 7f6bc2005640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-03T05:22:33.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:33.072+0000 7f6bc2005640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-03T05:22:33.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:22:33.072+0000 7f6bc2005640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-06-03T05:22:35.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-03T05:22:35.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-06-03T05:22:36.000 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:36.000 INFO:tasks.ceph:role: osd.4 2024-06-03T05:22:36.000 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:22:36.001 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-03T05:22:36.079 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T05:22:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T05:22:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T05:22:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T05:22:36.079 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T05:22:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-06-03T05:22:36.079 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T05:22:36.080 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T05:22:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T05:22:36.080 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T05:22:36.605 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-06-03T05:22:36.607 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-06-03T05:22:36.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-06-03T05:22:36.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-06-03T05:22:36.775 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-06-03T05:22:36.845 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:36.845 INFO:tasks.ceph:role: osd.5 2024-06-03T05:22:36.845 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:22:36.845 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-03T05:22:36.921 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T05:22:36.921 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T05:22:36.921 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T05:22:36.921 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T05:22:36.921 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T05:22:36.921 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-06-03T05:22:36.921 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T05:22:36.922 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T05:22:36.922 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T05:22:36.922 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T05:22:37.367 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-06-03T05:22:37.369 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-06-03T05:22:37.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-06-03T05:22:37.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-06-03T05:22:37.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-06-03T05:22:37.559 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:37.559 INFO:tasks.ceph:role: osd.6 2024-06-03T05:22:37.559 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:22:37.560 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-03T05:22:37.645 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T05:22:37.646 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T05:22:37.646 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T05:22:37.646 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T05:22:37.646 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T05:22:37.646 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-06-03T05:22:37.646 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T05:22:37.646 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T05:22:37.646 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T05:22:37.646 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T05:22:38.138 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-06-03T05:22:38.140 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-06-03T05:22:38.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-06-03T05:22:38.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-06-03T05:22:38.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-06-03T05:22:38.354 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T05:22:38.354 INFO:tasks.ceph:role: osd.7 2024-06-03T05:22:38.354 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:22:38.355 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-03T05:22:38.427 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T05:22:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T05:22:38.427 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T05:22:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T05:22:38.428 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T05:22:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-06-03T05:22:38.428 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T05:22:38.428 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T05:22:38.428 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T05:22:38.428 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T05:22:38.953 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-06-03T05:22:38.956 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-06-03T05:22:38.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-06-03T05:22:39.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-06-03T05:22:39.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:39.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:39.259+0000 7fdde9017640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-06-03T05:22:39.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:39.260+0000 7fdde9017640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-06-03T05:22:39.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:39.260+0000 7fdde9017640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-03T05:22:39.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:39.260+0000 7fdde9017640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-03T05:22:39.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:39.260+0000 7fdde9017640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-03T05:22:39.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:39.260+0000 7fdde9017640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-06-03T05:22:42.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-03T05:22:42.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:42.236+0000 7f30c9e0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-06-03T05:22:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:42.237+0000 7f30c9e0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-06-03T05:22:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:42.237+0000 7f30c9e0f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-03T05:22:42.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:42.237+0000 7f30c9e0f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-03T05:22:42.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:42.237+0000 7f30c9e0f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-03T05:22:42.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:42.237+0000 7f30c9e0f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-06-03T05:22:45.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-03T05:22:45.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:45.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:45.206+0000 7efe5a518640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-06-03T05:22:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:45.206+0000 7efe5a518640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-06-03T05:22:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:45.206+0000 7efe5a518640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-03T05:22:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:45.206+0000 7efe5a518640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-03T05:22:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:45.206+0000 7efe5a518640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-03T05:22:45.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:45.207+0000 7efe5a518640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-06-03T05:22:48.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-03T05:22:48.089 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:48.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:48.180+0000 7fc520585640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-06-03T05:22:48.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:48.180+0000 7fc520585640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-06-03T05:22:48.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:48.180+0000 7fc520585640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-03T05:22:48.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:48.181+0000 7fc520585640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-03T05:22:48.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:48.181+0000 7fc520585640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-03T05:22:48.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-06-03T05:22:48.181+0000 7fc520585640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-06-03T05:22:51.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-03T05:22:51.056 INFO:tasks.ceph:Reading keys from all nodes... 2024-06-03T05:22:51.056 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:51.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-06-03T05:22:51.088 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:51.088 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-06-03T05:22:51.159 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:51.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-06-03T05:22:51.229 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:51.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-06-03T05:22:51.297 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:51.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-06-03T05:22:51.365 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:51.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-06-03T05:22:51.396 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:51.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-06-03T05:22:51.467 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:51.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-06-03T05:22:51.537 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:51.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-06-03T05:22:51.605 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:51.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-06-03T05:22:51.673 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:22:51.673 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-06-03T05:22:51.692 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:22:51.692 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-06-03T05:22:51.731 INFO:tasks.ceph:Adding keys to all mons... 2024-06-03T05:22:51.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-03T05:22:51.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-03T05:22:51.774 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.y] 2024-06-03T05:22:51.774 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAJU11mXHRWGRAAnlBp1bT48sQjAgwuy+YFfw== 2024-06-03T05:22:51.774 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2024-06-03T05:22:51.774 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAQU11mb4QJBhAAeeAOumh5XOXLmxQZB64tVg== 2024-06-03T05:22:51.774 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2024-06-03T05:22:51.774 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQATU11moYi6BRAAQxeK84eKDxC9cHCniI9uNw== 2024-06-03T05:22:51.774 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2024-06-03T05:22:51.774 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAWU11myhoABRAAw0HW5NqlSZ6qF1EsJ6n6cQ== 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.3] 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAZU11mr5dfBBAABxiAylOMki7vcc7Q9czYUw== 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.x] 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAJU11mUdSAHRAAdcAUUuL656XO3bJaDZtqsA== 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.4] 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAfU11mYkiLDxAAJ73njX0KgRKuK/vsoIxKkQ== 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.5] 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAiU11mONorDhAA7HYMGdbNrcQjQK2zu6EeeA== 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.6] 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAlU11mkGBcDBAAmN3VpQwZBLqYVOeNmMPGow== 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.7] 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAoU11mXNfQChAACA6XtopWuNIwty/0JxLLzA== 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-06-03T05:22:51.775 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAJU11mClRFIBAAQaSUEWSyPgPJ75PIZ+OPkw== 2024-06-03T05:22:51.776 INFO:teuthology.orchestra.run.smithi038.stdout:[client.1] 2024-06-03T05:22:51.776 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAJU11mTEHPIxAA15oFdZtnbLA+YHHXBXtFNw== 2024-06-03T05:22:51.796 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2024-06-03T05:22:51.796 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAJU11mXHRWGRAAnlBp1bT48sQjAgwuy+YFfw== 2024-06-03T05:22:51.796 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-06-03T05:22:51.796 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAQU11mb4QJBhAAeeAOumh5XOXLmxQZB64tVg== 2024-06-03T05:22:51.796 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-06-03T05:22:51.796 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQATU11moYi6BRAAQxeK84eKDxC9cHCniI9uNw== 2024-06-03T05:22:51.796 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAWU11myhoABRAAw0HW5NqlSZ6qF1EsJ6n6cQ== 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAZU11mr5dfBBAABxiAylOMki7vcc7Q9czYUw== 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAJU11mUdSAHRAAdcAUUuL656XO3bJaDZtqsA== 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAfU11mYkiLDxAAJ73njX0KgRKuK/vsoIxKkQ== 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAiU11mONorDhAA7HYMGdbNrcQjQK2zu6EeeA== 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAlU11mkGBcDBAAmN3VpQwZBLqYVOeNmMPGow== 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAoU11mXNfQChAACA6XtopWuNIwty/0JxLLzA== 2024-06-03T05:22:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-06-03T05:22:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAJU11mClRFIBAAQaSUEWSyPgPJ75PIZ+OPkw== 2024-06-03T05:22:51.798 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2024-06-03T05:22:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAJU11mTEHPIxAA15oFdZtnbLA+YHHXBXtFNw== 2024-06-03T05:22:51.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-03T05:22:51.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-03T05:22:51.892 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:51.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:51.982 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:51.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.090 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-03T05:22:52.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-03T05:22:52.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T05:22:52.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-03T05:22:52.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-03T05:22:52.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-03T05:22:52.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-03T05:22:52.534 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-06-03T05:22:52.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-06-03T05:22:52.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-03T05:22:52.662 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-06-03T05:22:52.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-06-03T05:22:52.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-03T05:22:52.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-06-03T05:22:52.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-06-03T05:22:52.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-03T05:22:53.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-06-03T05:22:53.034 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:53.037 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-03T05:22:53.090 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-06-03T05:22:53.090 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-06-03T05:22:53.090 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-03T05:22:53.091 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-06-03T05:22:53.093 INFO:tasks.ceph.mon.a:Started 2024-06-03T05:22:53.093 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-06-03T05:22:53.093 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-03T05:22:53.093 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-06-03T05:22:53.095 INFO:tasks.ceph.mon.c:Started 2024-06-03T05:22:53.095 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-06-03T05:22:53.096 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-03T05:22:53.096 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-06-03T05:22:53.132 INFO:tasks.ceph.mon.b:Started 2024-06-03T05:22:53.132 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-06-03T05:22:53.132 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-06-03T05:22:53.132 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-06-03T05:22:53.134 INFO:tasks.ceph.mgr.y:Started 2024-06-03T05:22:53.134 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-06-03T05:22:53.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-06-03T05:22:53.136 INFO:tasks.ceph.mgr.x:Started 2024-06-03T05:22:53.137 INFO:tasks.ceph:Setting crush tunables to default 2024-06-03T05:22:53.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd crush tunables default 2024-06-03T05:23:00.598 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:07.397 35035== Warning: unimplemented fcntl command: 1036 2024-06-03T05:23:00.598 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:07.397 35037== Warning: unimplemented fcntl command: 1036 2024-06-03T05:23:00.607 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:07.352 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:23:00.925 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:07.724 35035== Warning: unimplemented fcntl command: 1036 2024-06-03T05:23:00.926 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:07.725 35037== Warning: unimplemented fcntl command: 1036 2024-06-03T05:23:00.933 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:07.678 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:23:11.465 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:11.464+0000 7f23f7414040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-03T05:23:11.492 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:11.492+0000 7f77cd0fa040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-03T05:23:11.625 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:11.625+0000 7f23f7414040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-03T05:23:11.666 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:11.666+0000 7f77cd0fa040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-03T05:23:11.686 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:11.682+0000 7f23f7414040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-03T05:23:11.727 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:11.728+0000 7f77cd0fa040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-03T05:23:11.952 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:11.953+0000 7f23f7414040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-03T05:23:12.002 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:12.002+0000 7f23f7414040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-03T05:23:12.012 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:12.013+0000 7f77cd0fa040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-03T05:23:12.014 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-06-03T05:23:12.026 INFO:tasks.ceph:check_enable_crimson: False 2024-06-03T05:23:12.026 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-06-03T05:23:12.027 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:23:12.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-06-03T05:23:12.053 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:23:12.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-06-03T05:23:12.068 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:12.068+0000 7f77cd0fa040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-03T05:23:12.097 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:23:12.097 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-06-03T05:23:12.164 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-06-03T05:23:12.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-06-03T05:23:12.230 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:23:12.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-06-03T05:23:12.257 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:23:12.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-06-03T05:23:12.323 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:23:12.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-06-03T05:23:12.389 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-06-03T05:23:12.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-06-03T05:23:12.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 32b256d9-eee7-435b-8aec-fef2ba90366c 0 2024-06-03T05:23:12.842 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:12.842+0000 7f77cd0fa040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-03T05:23:12.850 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:12.850+0000 7f23f7414040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-03T05:23:12.942 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:12.942+0000 7f77cd0fa040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-03T05:23:12.942 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-06-03T05:23:12.947 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:12.948+0000 7f23f7414040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-03T05:23:12.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 371c2e00-54db-430b-8497-a4c9ff6753b9 1 2024-06-03T05:23:12.997 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:12.998+0000 7f23f7414040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-03T05:23:12.999 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:13.000+0000 7f77cd0fa040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-03T05:23:13.177 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:13.177+0000 7f23f7414040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-03T05:23:13.182 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:13.183+0000 7f77cd0fa040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-03T05:23:13.229 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:13.230+0000 7f23f7414040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-03T05:23:13.240 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:13.240+0000 7f77cd0fa040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-03T05:23:13.284 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:13.285+0000 7f23f7414040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-03T05:23:13.296 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:13.296+0000 7f77cd0fa040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-03T05:23:13.404 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:13.404+0000 7f23f7414040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-03T05:23:13.415 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:13.414+0000 7f77cd0fa040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-03T05:23:13.498 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-06-03T05:23:13.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new e8145335-ec5d-4dd1-9250-a4ab5b159d29 2 2024-06-03T05:23:13.822 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:13.822+0000 7f77cd0fa040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-03T05:23:13.834 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:13.835+0000 7f23f7414040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-03T05:23:13.990 INFO:teuthology.orchestra.run.smithi064.stdout:2 2024-06-03T05:23:14.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c 3 2024-06-03T05:23:14.103 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:14.103+0000 7f77cd0fa040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-03T05:23:14.120 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:14.120+0000 7f23f7414040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-03T05:23:14.163 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:14.162+0000 7f77cd0fa040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-03T05:23:14.179 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:14.179+0000 7f23f7414040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-03T05:23:14.285 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:14.285+0000 7f77cd0fa040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-03T05:23:14.304 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:14.304+0000 7f23f7414040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-03T05:23:14.398 INFO:teuthology.orchestra.run.smithi064.stdout:3 2024-06-03T05:23:14.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 46b46f54-04d0-46c0-809c-64fa385e6780 4 2024-06-03T05:23:14.591 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:14.591+0000 7f77cd0fa040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-03T05:23:14.609 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:14.609+0000 7f23f7414040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-03T05:23:14.641 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:14.641+0000 7f77cd0fa040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-03T05:23:14.659 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:14.660+0000 7f23f7414040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-03T05:23:14.784 INFO:teuthology.orchestra.run.smithi064.stdout:4 2024-06-03T05:23:14.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new f7a50ac0-eba9-4602-b90a-2dbc4bb20593 5 2024-06-03T05:23:14.810 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:14.809+0000 7f77cd0fa040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-03T05:23:14.831 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:14.831+0000 7f23f7414040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-03T05:23:14.961 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:14.958+0000 7f77cd0fa040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-03T05:23:14.980 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:14.980+0000 7f23f7414040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-03T05:23:15.011 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:15.010+0000 7f77cd0fa040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-03T05:23:15.032 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:15.033+0000 7f23f7414040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-03T05:23:15.084 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:15.084+0000 7f77cd0fa040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-03T05:23:15.107 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:15.108+0000 7f23f7414040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-03T05:23:15.137 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:15.136+0000 7f77cd0fa040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-03T05:23:15.159 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:15.159+0000 7f23f7414040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-03T05:23:15.235 INFO:teuthology.orchestra.run.smithi064.stdout:5 2024-06-03T05:23:15.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 081024c0-2482-4846-a7bd-594a58f2ceef 6 2024-06-03T05:23:15.646 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:15.645+0000 7f77cd0fa040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-03T05:23:15.667 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:15.667+0000 7f23f7414040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-03T05:23:15.669 INFO:teuthology.orchestra.run.smithi064.stdout:6 2024-06-03T05:23:15.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 6234c757-8f7b-497e-9347-4a7983df592c 7 2024-06-03T05:23:15.769 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:15.769+0000 7f77cd0fa040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-03T05:23:15.792 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-06-03T05:23:15.793+0000 7f23f7414040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-03T05:23:16.309 INFO:teuthology.orchestra.run.smithi064.stdout:7 2024-06-03T05:23:16.320 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T05:23:16.320 INFO:tasks.ceph.osd.0:Restarting daemon 2024-06-03T05:23:16.320 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-06-03T05:23:16.323 INFO:tasks.ceph.osd.0:Started 2024-06-03T05:23:16.323 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T05:23:16.323 INFO:tasks.ceph.osd.1:Restarting daemon 2024-06-03T05:23:16.323 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-06-03T05:23:16.325 INFO:tasks.ceph.osd.1:Started 2024-06-03T05:23:16.325 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T05:23:16.325 INFO:tasks.ceph.osd.2:Restarting daemon 2024-06-03T05:23:16.325 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-06-03T05:23:16.327 INFO:tasks.ceph.osd.2:Started 2024-06-03T05:23:16.328 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T05:23:16.328 INFO:tasks.ceph.osd.3:Restarting daemon 2024-06-03T05:23:16.328 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-06-03T05:23:16.330 INFO:tasks.ceph.osd.3:Started 2024-06-03T05:23:16.330 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T05:23:16.330 INFO:tasks.ceph.osd.4:Restarting daemon 2024-06-03T05:23:16.330 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-06-03T05:23:16.333 INFO:tasks.ceph.osd.4:Started 2024-06-03T05:23:16.333 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T05:23:16.333 INFO:tasks.ceph.osd.5:Restarting daemon 2024-06-03T05:23:16.334 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-06-03T05:23:16.336 INFO:tasks.ceph.osd.5:Started 2024-06-03T05:23:16.336 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T05:23:16.336 INFO:tasks.ceph.osd.6:Restarting daemon 2024-06-03T05:23:16.336 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-06-03T05:23:16.338 INFO:tasks.ceph.osd.6:Started 2024-06-03T05:23:16.338 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T05:23:16.338 INFO:tasks.ceph.osd.7:Restarting daemon 2024-06-03T05:23:16.338 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-06-03T05:23:16.341 INFO:tasks.ceph.osd.7:Started 2024-06-03T05:23:16.341 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T05:23:16.910 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:23:16.910 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":10,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:23:16.043190+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:23:16.927 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-06-03T05:23:16.927 INFO:tasks.ceph:Waiting for OSDs to come up 2024-06-03T05:23:17.734 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T05:23:17.906 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:17.905+0000 7f776e4f5640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-06-03T05:23:17.931 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-06-03T05:23:17.930+0000 7f776e4f5640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-03T05:23:18.932 INFO:teuthology.misc.health.smithi038.stdout: 2024-06-03T05:23:18.933 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":10,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:23:16.043190+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:23:18.949 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T05:23:23.995 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:07.592 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.060 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:07.655 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.061 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:07.658 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.122 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:07.718 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.209 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:07.793 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.269 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:07.854 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.300 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:07.884 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.332 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:07.930 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.363 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:07.946 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.393 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:07.991 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.439 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:08.011 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.499 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:08.071 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.532 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:08.137 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.537 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:08.122 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.593 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:08.198 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.597 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:08.182 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:24.604 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:23:24.598+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:23:24.692 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:23:24.686+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:23:24.809 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:23:24.804+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:23:24.919 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:23:24.912+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:23:24.940 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:23:24.933+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:23:25.055 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:08.652 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.064 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:23:25.058+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:23:25.135 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:08.731 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.189 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:23:25.180+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:23:25.191 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:23:25.184+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:23:25.256 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:08.841 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.363 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:08.947 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.380 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:08.978 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.429 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:09.026 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.503 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:09.076 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.512 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:09.108 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.629 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:09.214 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.709 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:09.239 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.709 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:09.221 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.737 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:09.320 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.746 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:09.343 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.755 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:09.353 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.809 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T05:23:25.835 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:09.431 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:25.871 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:09.444 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.009 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:09.614 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.010 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:09.594 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.011 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:09.608 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.017 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:09.602 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.095 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:09.679 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.100 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:09.696 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.121 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:09.719 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.249 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:09.821 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.282 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:09.879 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.324 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:09.909 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.326 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:09.931 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.369 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:09.954 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.372 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:09.968 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.386 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:09.970 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.411 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:10.009 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.493 INFO:teuthology.misc.health.smithi038.stdout: 2024-06-03T05:23:26.493 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":10,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:23:16.043190+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:23:26.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T05:23:26.542 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:10.114 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.548 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:10.145 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.591 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:10.196 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.611 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:10.196 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.640 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:10.236 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.684 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:10.268 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.689 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:10.275 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.709 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:10.308 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.813 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:10.410 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.838 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:10.411 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.861 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:10.466 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.905 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:10.490 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.909 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:10.505 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.984 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:10.567 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.987 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:10.573 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:26.988 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:10.586 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.078 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:10.675 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.129 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:10.734 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.134 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:10.707 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.175 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:10.771 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.199 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:10.783 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.277 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:10.862 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.299 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:10.883 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.309 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:10.907 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.350 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:10.947 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.394 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:10.999 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.422 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:10.995 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.451 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:11.047 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.489 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:11.073 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.566 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:11.152 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.611 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:11.195 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.615 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:11.213 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.664 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:11.269 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.717 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:11.290 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.782 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:11.366 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.844 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:11.441 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.861 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:11.445 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.900 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:11.497 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.933 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:11.516 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.934 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:11.530 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.936 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:11.541 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.936 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:11.535 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:27.991 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:11.587 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.021 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:11.593 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.154 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:11.739 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.310 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:11.894 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.384 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:11.968 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.415 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:12.020 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.463 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:12.047 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.468 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:12.066 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.492 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:12.097 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.521 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:12.104 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.525 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:12.124 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.576 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:12.149 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.633 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:12.206 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.747 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:12.333 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:28.818 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:12.404 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:33.311 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T05:23:33.488 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:17.084 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:33.938 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:17.537 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:33.943 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:17.540 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:34.044 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:17.628 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:34.150 INFO:teuthology.misc.health.smithi038.stdout: 2024-06-03T05:23:34.151 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":10,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:23:16.043190+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:23:34.151 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:17.734 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:34.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T05:23:34.485 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:18.058 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:34.515 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:18.120 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:34.703 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:18.289 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.170 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:22.764 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.172 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:22.768 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.564 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:23.161 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.569 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:23.166 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.797 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:23.396 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.801 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:23.400 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.865 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:23.449 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.871 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:23.456 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.976 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:23.560 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:39.981 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:23.564 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:40.625 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:24.198 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:40.632 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:24.204 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:40.652 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:24.237 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:40.658 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:24.244 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:40.845 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:24.450 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:40.850 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:24.455 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:40.975 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T05:23:41.858 INFO:teuthology.misc.health.smithi038.stdout: 2024-06-03T05:23:41.858 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":10,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:23:16.043190+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:23:41.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T05:23:44.057 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:27.653 35183== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:44.322 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:27.919 35182== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:44.345 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:27.930 35184== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:44.622 INFO:tasks.ceph.osd.2.smithi038.stderr:==00:00:00:28.220 35352== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:44.658 INFO:tasks.ceph.osd.3.smithi038.stderr:==00:00:00:28.242 35354== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:44.917 INFO:tasks.ceph.osd.7.smithi064.stderr:==00:00:00:28.490 35185== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:45.168 INFO:tasks.ceph.osd.1.smithi038.stderr:==00:00:00:28.754 35353== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:45.686 INFO:tasks.ceph.osd.0.smithi038.stderr:==00:00:00:29.291 35351== Warning: unimplemented fcntl command: 1038 2024-06-03T05:23:48.680 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T05:23:49.400 INFO:teuthology.misc.health.smithi038.stdout: 2024-06-03T05:23:49.400 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":10,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:23:16.043190+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:23:49.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T05:23:52.150 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:23:52.147+0000 5dce500 -1 osd.6 0 log_to_monitors true 2024-06-03T05:23:52.418 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:23:52.411+0000 5dce500 -1 osd.5 0 log_to_monitors true 2024-06-03T05:23:52.807 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:23:52.802+0000 5dce500 -1 osd.4 0 log_to_monitors true 2024-06-03T05:23:52.945 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:23:52.893+0000 5dce500 -1 osd.7 0 log_to_monitors true 2024-06-03T05:23:53.051 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:23:53.049+0000 5dce500 -1 osd.2 0 log_to_monitors true 2024-06-03T05:23:53.055 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:23:53.053+0000 5dce500 -1 osd.3 0 log_to_monitors true 2024-06-03T05:23:53.558 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:23:53.555+0000 5dce500 -1 osd.1 0 log_to_monitors true 2024-06-03T05:23:54.146 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:23:54.059+0000 5dce500 -1 osd.0 0 log_to_monitors true 2024-06-03T05:23:55.508 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:23:55.503+0000 c49a640 -1 osd.6 0 waiting for initial osdmap 2024-06-03T05:23:55.508 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:23:55.504+0000 c49a640 -1 osd.5 0 waiting for initial osdmap 2024-06-03T05:23:55.893 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:23:55.872+0000 10260640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:23:55.951 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:23:55.937+0000 10260640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:23:56.220 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T05:23:56.977 INFO:teuthology.misc.health.smithi038.stdout: 2024-06-03T05:23:56.978 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":13,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:23:56.154143+0000","last_up_change":"2024-06-03T05:23:56.154143+0000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3915827095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":3915827095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":3915827095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3915827095}]},"public_addr":"172.21.15.64:6800/3915827095","cluster_addr":"172.21.15.64:6801/3915827095","heartbeat_back_addr":"172.21.15.64:6803/3915827095","heartbeat_front_addr":"172.21.15.64:6802/3915827095","state":["exists","up"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1144336651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1144336651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6807","nonce":1144336651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1144336651}]},"public_addr":"172.21.15.64:6804/1144336651","cluster_addr":"172.21.15.64:6805/1144336651","heartbeat_back_addr":"172.21.15.64:6807/1144336651","heartbeat_front_addr":"172.21.15.64:6806/1144336651","state":["exists","up"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:23:56.990 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-06-03T05:23:57.884 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:23:57.881+0000 c49a640 -1 osd.0 0 waiting for initial osdmap 2024-06-03T05:23:58.511 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:23:58.498+0000 10260640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:24:03.795 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T05:24:04.823 INFO:teuthology.misc.health.smithi038.stdout: 2024-06-03T05:24:04.823 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":17,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:24:01.101914+0000","last_up_change":"2024-06-03T05:23:59.524891+0000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T05:23:56.753149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":905141127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":905141127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":905141127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":905141127}]},"public_addr":"172.21.15.38:6809/905141127","cluster_addr":"172.21.15.38:6811/905141127","heartbeat_back_addr":"172.21.15.38:6816/905141127","heartbeat_front_addr":"172.21.15.38:6813/905141127","state":["exists","up"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3915827095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":3915827095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":3915827095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3915827095}]},"public_addr":"172.21.15.64:6800/3915827095","cluster_addr":"172.21.15.64:6801/3915827095","heartbeat_back_addr":"172.21.15.64:6803/3915827095","heartbeat_front_addr":"172.21.15.64:6802/3915827095","state":["exists","up"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1144336651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1144336651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6807","nonce":1144336651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1144336651}]},"public_addr":"172.21.15.64:6804/1144336651","cluster_addr":"172.21.15.64:6805/1144336651","heartbeat_back_addr":"172.21.15.64:6807/1144336651","heartbeat_front_addr":"172.21.15.64:6806/1144336651","state":["exists","up"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:56.081253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.533669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.310102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:24:04.844 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-06-03T05:24:04.855 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:04.854+0000 5dce500 -1 bdev(0x164e0ef0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-06-03T05:24:05.019 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:04.962+0000 5dce500 -1 bdev(0x164e0ef0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-06-03T05:24:05.045 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:05.019+0000 5dce500 -1 bdev(0x164e0ef0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-06-03T05:24:05.089 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:04.925+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.090 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:04.944+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-06-03T05:24:05.315 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.053+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.346 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.289+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:05.377 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.364+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-06-03T05:24:05.381 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.093+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.390 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.381+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.391 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.390+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.408 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:05.050+0000 5dce500 -1 bdev(0x164e0ef0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-06-03T05:24:05.423 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.378+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:05.459 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.422+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.459 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.441+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.489 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.460+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.517 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:05.357+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 3 2024-06-03T05:24:05.530 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:05.378+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:05.531 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:05.479+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:05.531 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:05.505+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:05.544 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:05.518+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:05.565 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.489+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.682 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.424+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:05.684 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.659+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:05.713 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.684+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:05.739 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.713+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:05.775 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:05.461+0000 5dce500 -1 bdev(0x1966c830 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-06-03T05:24:05.826 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.727+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:05.983 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.552+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:05.991 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.839+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 4 2024-06-03T05:24:05.991 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.979+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:06.031 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:05.992+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:06.051 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:06.038+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-06-03T05:24:06.074 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.601+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-06-03T05:24:06.075 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:05.985+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:06.075 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:06.037+0000 5dce500 -1 bdev(0x164cef70 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-06-03T05:24:06.135 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:05.531+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:06.138 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:05.553+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:06.163 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:05.780+0000 5dce500 -1 bdev(0x1966c830 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-06-03T05:24:06.163 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:05.815+0000 5dce500 -1 bdev(0x1966c830 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-06-03T05:24:06.176 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:06.140+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-06-03T05:24:06.189 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:06.164+0000 5dce500 -1 bdev(0x1966c830 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-06-03T05:24:06.229 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:06.190+0000 5dce500 -1 bdev(0x1966c830 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 2 2024-06-03T05:24:06.249 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:06.122+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 4 2024-06-03T05:24:06.249 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:06.174+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:06.345 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:06.177+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:06.384 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:06.229+0000 5dce500 -1 bdev(0x1966c830 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 2 2024-06-03T05:24:06.384 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:06.384+0000 5dce500 -1 bdev(0x1966c830 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-06-03T05:24:06.396 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:06.262+0000 5dce500 -1 bdev(0x164cc510 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T05:24:06.510 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:06.399+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:06.524 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:06.511+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:06.527 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:06.503+0000 5dce500 -1 bdev(0x1966c830 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-06-03T05:24:06.539 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:06.525+0000 5dce500 -1 bdev(0x164a76c0 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-06-03T05:24:07.078 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:07.073+0000 c49a640 -1 osd.7 0 waiting for initial osdmap 2024-06-03T05:24:07.086 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:07.083+0000 ac84640 -1 osd.4 0 waiting for initial osdmap 2024-06-03T05:24:07.428 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:07.422+0000 c49a640 -1 osd.2 0 waiting for initial osdmap 2024-06-03T05:24:07.517 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:07.501+0000 14268640 -1 osd.4 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:24:07.542 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:07.529+0000 14268640 -1 osd.7 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:24:07.638 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:07.635+0000 c49a640 -1 osd.3 0 waiting for initial osdmap 2024-06-03T05:24:07.707 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:07.704+0000 c49a640 -1 osd.1 0 waiting for initial osdmap 2024-06-03T05:24:07.905 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:14.704 35035== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:07.930 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:07.916+0000 14268640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:24:08.001 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:14.746 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:08.027 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:14.826 35035== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:08.062 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:14.806 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:08.091 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:14.890 35037== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:08.092 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:08.072+0000 14268640 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:24:08.179 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:14.978 35037== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:08.184 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:08.169+0000 14268640 -1 osd.1 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:24:08.321 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:15.066 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:08.350 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:15.149 35035== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:08.479 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:15.278 35037== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:11.651 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T05:24:12.273 INFO:teuthology.misc.health.smithi038.stdout: 2024-06-03T05:24:12.273 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":20,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:24:10.948450+0000","last_up_change":"2024-06-03T05:24:09.927340+0000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T05:23:56.753149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":905141127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":905141127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":905141127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":905141127}]},"public_addr":"172.21.15.38:6809/905141127","cluster_addr":"172.21.15.38:6811/905141127","heartbeat_back_addr":"172.21.15.38:6816/905141127","heartbeat_front_addr":"172.21.15.38:6813/905141127","state":["exists","up"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3490615740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3490615740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3490615740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3490615740}]},"public_addr":"172.21.15.38:6810/3490615740","cluster_addr":"172.21.15.38:6812/3490615740","heartbeat_back_addr":"172.21.15.38:6815/3490615740","heartbeat_front_addr":"172.21.15.38:6814/3490615740","state":["exists","up"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":3359326199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3359326199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3359326199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":3359326199}]},"public_addr":"172.21.15.38:6805/3359326199","cluster_addr":"172.21.15.38:6806/3359326199","heartbeat_back_addr":"172.21.15.38:6808/3359326199","heartbeat_front_addr":"172.21.15.38:6807/3359326199","state":["exists","up"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":2289724788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2289724788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2289724788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2289724788}]},"public_addr":"172.21.15.38:6801/2289724788","cluster_addr":"172.21.15.38:6802/2289724788","heartbeat_back_addr":"172.21.15.38:6804/2289724788","heartbeat_front_addr":"172.21.15.38:6803/2289724788","state":["exists","up"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3833397294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":3833397294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":3833397294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3833397294}]},"public_addr":"172.21.15.64:6808/3833397294","cluster_addr":"172.21.15.64:6809/3833397294","heartbeat_back_addr":"172.21.15.64:6811/3833397294","heartbeat_front_addr":"172.21.15.64:6810/3833397294","state":["exists","up"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3915827095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":3915827095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":3915827095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3915827095}]},"public_addr":"172.21.15.64:6800/3915827095","cluster_addr":"172.21.15.64:6801/3915827095","heartbeat_back_addr":"172.21.15.64:6803/3915827095","heartbeat_front_addr":"172.21.15.64:6802/3915827095","state":["exists","up"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1144336651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1144336651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6807","nonce":1144336651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1144336651}]},"public_addr":"172.21.15.64:6804/1144336651","cluster_addr":"172.21.15.64:6805/1144336651","heartbeat_back_addr":"172.21.15.64:6807/1144336651","heartbeat_front_addr":"172.21.15.64:6806/1144336651","state":["exists","up"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":370645108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":370645108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":370645108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":370645108}]},"public_addr":"172.21.15.64:6812/370645108","cluster_addr":"172.21.15.64:6813/370645108","heartbeat_back_addr":"172.21.15.64:6815/370645108","heartbeat_front_addr":"172.21.15.64:6814/370645108","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:56.081253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.642638+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.131979+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.147738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.907705+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.533669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.310102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.959957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:24:12.285 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-06-03T05:24:12.286 INFO:tasks.ceph:Creating RBD pool 2024-06-03T05:24:12.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd pool create rbd 8 2024-06-03T05:24:13.307 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'rbd' created 2024-06-03T05:24:13.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-06-03T05:24:14.391 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rbd' on pool 'rbd' 2024-06-03T05:24:14.407 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-06-03T05:24:14.408 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-03T05:24:14.408 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-06-03T05:24:14.408 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-06-03T05:24:14.408 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-06-03T05:24:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:15.242 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":3846177730}]},"active_addr":"172.21.15.38:6800/3846177730","active_change":"2024-06-03T05:23:15.995561+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3529145374}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1302473185}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1314613108}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":589119403}]}]} 2024-06-03T05:24:15.246 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-06-03T05:24:15.246 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-06-03T05:24:15.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T05:24:15.966 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:15.966 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":23,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:24:15.314775+0000","last_up_change":"2024-06-03T05:24:09.927340+0000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T05:23:56.753149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-03T05:24:12.898324+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":905141127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":905141127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":905141127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":905141127}]},"public_addr":"172.21.15.38:6809/905141127","cluster_addr":"172.21.15.38:6811/905141127","heartbeat_back_addr":"172.21.15.38:6816/905141127","heartbeat_front_addr":"172.21.15.38:6813/905141127","state":["exists","up"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3490615740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3490615740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3490615740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3490615740}]},"public_addr":"172.21.15.38:6810/3490615740","cluster_addr":"172.21.15.38:6812/3490615740","heartbeat_back_addr":"172.21.15.38:6815/3490615740","heartbeat_front_addr":"172.21.15.38:6814/3490615740","state":["exists","up"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":3359326199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3359326199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3359326199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":3359326199}]},"public_addr":"172.21.15.38:6805/3359326199","cluster_addr":"172.21.15.38:6806/3359326199","heartbeat_back_addr":"172.21.15.38:6808/3359326199","heartbeat_front_addr":"172.21.15.38:6807/3359326199","state":["exists","up"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":2289724788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2289724788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2289724788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2289724788}]},"public_addr":"172.21.15.38:6801/2289724788","cluster_addr":"172.21.15.38:6802/2289724788","heartbeat_back_addr":"172.21.15.38:6804/2289724788","heartbeat_front_addr":"172.21.15.38:6803/2289724788","state":["exists","up"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3833397294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":3833397294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":3833397294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3833397294}]},"public_addr":"172.21.15.64:6808/3833397294","cluster_addr":"172.21.15.64:6809/3833397294","heartbeat_back_addr":"172.21.15.64:6811/3833397294","heartbeat_front_addr":"172.21.15.64:6810/3833397294","state":["exists","up"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3915827095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":3915827095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":3915827095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3915827095}]},"public_addr":"172.21.15.64:6800/3915827095","cluster_addr":"172.21.15.64:6801/3915827095","heartbeat_back_addr":"172.21.15.64:6803/3915827095","heartbeat_front_addr":"172.21.15.64:6802/3915827095","state":["exists","up"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1144336651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1144336651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6807","nonce":1144336651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1144336651}]},"public_addr":"172.21.15.64:6804/1144336651","cluster_addr":"172.21.15.64:6805/1144336651","heartbeat_back_addr":"172.21.15.64:6807/1144336651","heartbeat_front_addr":"172.21.15.64:6806/1144336651","state":["exists","up"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":370645108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":370645108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":370645108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":370645108}]},"public_addr":"172.21.15.64:6812/370645108","cluster_addr":"172.21.15.64:6813/370645108","heartbeat_back_addr":"172.21.15.64:6815/370645108","heartbeat_front_addr":"172.21.15.64:6814/370645108","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:56.081253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.642638+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.131979+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.147738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.907705+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.533669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.310102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.959957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:24:15.979 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-06-03T05:24:15.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T05:24:16.697 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:16.697 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":24,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:24:16.465229+0000","last_up_change":"2024-06-03T05:24:09.927340+0000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T05:23:56.753149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-03T05:24:12.898324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":905141127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":905141127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":905141127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":905141127}]},"public_addr":"172.21.15.38:6809/905141127","cluster_addr":"172.21.15.38:6811/905141127","heartbeat_back_addr":"172.21.15.38:6816/905141127","heartbeat_front_addr":"172.21.15.38:6813/905141127","state":["exists","up"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3490615740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3490615740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3490615740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3490615740}]},"public_addr":"172.21.15.38:6810/3490615740","cluster_addr":"172.21.15.38:6812/3490615740","heartbeat_back_addr":"172.21.15.38:6815/3490615740","heartbeat_front_addr":"172.21.15.38:6814/3490615740","state":["exists","up"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":3359326199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3359326199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3359326199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":3359326199}]},"public_addr":"172.21.15.38:6805/3359326199","cluster_addr":"172.21.15.38:6806/3359326199","heartbeat_back_addr":"172.21.15.38:6808/3359326199","heartbeat_front_addr":"172.21.15.38:6807/3359326199","state":["exists","up"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":2289724788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2289724788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2289724788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2289724788}]},"public_addr":"172.21.15.38:6801/2289724788","cluster_addr":"172.21.15.38:6802/2289724788","heartbeat_back_addr":"172.21.15.38:6804/2289724788","heartbeat_front_addr":"172.21.15.38:6803/2289724788","state":["exists","up"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3833397294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":3833397294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":3833397294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3833397294}]},"public_addr":"172.21.15.64:6808/3833397294","cluster_addr":"172.21.15.64:6809/3833397294","heartbeat_back_addr":"172.21.15.64:6811/3833397294","heartbeat_front_addr":"172.21.15.64:6810/3833397294","state":["exists","up"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3915827095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":3915827095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":3915827095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3915827095}]},"public_addr":"172.21.15.64:6800/3915827095","cluster_addr":"172.21.15.64:6801/3915827095","heartbeat_back_addr":"172.21.15.64:6803/3915827095","heartbeat_front_addr":"172.21.15.64:6802/3915827095","state":["exists","up"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1144336651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1144336651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6807","nonce":1144336651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1144336651}]},"public_addr":"172.21.15.64:6804/1144336651","cluster_addr":"172.21.15.64:6805/1144336651","heartbeat_back_addr":"172.21.15.64:6807/1144336651","heartbeat_front_addr":"172.21.15.64:6806/1144336651","state":["exists","up"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":370645108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":370645108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":370645108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":370645108}]},"public_addr":"172.21.15.64:6812/370645108","cluster_addr":"172.21.15.64:6813/370645108","heartbeat_back_addr":"172.21.15.64:6815/370645108","heartbeat_front_addr":"172.21.15.64:6814/370645108","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:56.081253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.642638+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.131979+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.147738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.907705+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.533669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.310102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.959957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:24:16.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-06-03T05:24:16.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-06-03T05:24:16.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-06-03T05:24:16.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-06-03T05:24:16.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-06-03T05:24:16.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-06-03T05:24:16.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-06-03T05:24:16.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-06-03T05:24:17.136 INFO:teuthology.orchestra.run.smithi038.stdout:68719476743 2024-06-03T05:24:17.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-03T05:24:17.146 INFO:teuthology.orchestra.run.smithi038.stdout:77309411338 2024-06-03T05:24:17.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-03T05:24:17.169 INFO:teuthology.orchestra.run.smithi038.stdout:81604378634 2024-06-03T05:24:17.170 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-03T05:24:17.204 INFO:teuthology.orchestra.run.smithi038.stdout:55834574854 2024-06-03T05:24:17.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-06-03T05:24:17.233 INFO:teuthology.orchestra.run.smithi038.stdout:77309411338 2024-06-03T05:24:17.233 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-03T05:24:17.236 INFO:teuthology.orchestra.run.smithi038.stdout:77309411336 2024-06-03T05:24:17.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-03T05:24:17.237 INFO:teuthology.orchestra.run.smithi038.stdout:55834574855 2024-06-03T05:24:17.237 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-03T05:24:17.247 INFO:teuthology.orchestra.run.smithi038.stdout:77309411335 2024-06-03T05:24:17.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-03T05:24:18.051 INFO:teuthology.orchestra.run.smithi038.stdout:77309411337 2024-06-03T05:24:18.065 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411337 for osd.4 2024-06-03T05:24:18.183 INFO:teuthology.orchestra.run.smithi038.stdout:68719476742 2024-06-03T05:24:18.193 INFO:teuthology.orchestra.run.smithi038.stdout:81604378633 2024-06-03T05:24:18.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476742 for osd.0 2024-06-03T05:24:18.214 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378634 got 81604378633 for osd.3 2024-06-03T05:24:18.281 INFO:teuthology.orchestra.run.smithi038.stdout:77309411337 2024-06-03T05:24:18.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411337 for osd.1 2024-06-03T05:24:18.626 INFO:teuthology.orchestra.run.smithi038.stdout:55834574852 2024-06-03T05:24:18.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.6 2024-06-03T05:24:18.686 INFO:teuthology.orchestra.run.smithi038.stdout:77309411334 2024-06-03T05:24:18.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411335 got 77309411334 for osd.7 2024-06-03T05:24:18.755 INFO:teuthology.orchestra.run.smithi038.stdout:77309411335 2024-06-03T05:24:18.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411336 got 77309411335 for osd.2 2024-06-03T05:24:18.871 INFO:teuthology.orchestra.run.smithi038.stdout:55834574853 2024-06-03T05:24:18.885 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.5 2024-06-03T05:24:19.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-03T05:24:19.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-03T05:24:19.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-03T05:24:19.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-03T05:24:19.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-06-03T05:24:19.706 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-03T05:24:19.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-03T05:24:19.872 INFO:teuthology.orchestra.run.smithi038.stdout:77309411337 2024-06-03T05:24:19.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-03T05:24:19.891 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411337 for osd.4 2024-06-03T05:24:20.199 INFO:teuthology.orchestra.run.smithi038.stdout:68719476743 2024-06-03T05:24:20.214 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.0 2024-06-03T05:24:20.214 DEBUG:teuthology.parallel:result is None 2024-06-03T05:24:20.230 INFO:teuthology.orchestra.run.smithi038.stdout:77309411338 2024-06-03T05:24:20.248 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411338 for osd.1 2024-06-03T05:24:20.248 DEBUG:teuthology.parallel:result is None 2024-06-03T05:24:20.294 INFO:teuthology.orchestra.run.smithi038.stdout:81604378634 2024-06-03T05:24:20.311 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378634 got 81604378634 for osd.3 2024-06-03T05:24:20.311 DEBUG:teuthology.parallel:result is None 2024-06-03T05:24:20.532 INFO:teuthology.orchestra.run.smithi038.stdout:77309411336 2024-06-03T05:24:20.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411336 got 77309411336 for osd.2 2024-06-03T05:24:20.546 DEBUG:teuthology.parallel:result is None 2024-06-03T05:24:20.626 INFO:teuthology.orchestra.run.smithi038.stdout:77309411335 2024-06-03T05:24:20.631 INFO:teuthology.orchestra.run.smithi038.stdout:55834574854 2024-06-03T05:24:20.642 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411335 got 77309411335 for osd.7 2024-06-03T05:24:20.642 DEBUG:teuthology.parallel:result is None 2024-06-03T05:24:20.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2024-06-03T05:24:20.646 DEBUG:teuthology.parallel:result is None 2024-06-03T05:24:20.822 INFO:teuthology.orchestra.run.smithi038.stdout:55834574855 2024-06-03T05:24:20.835 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-06-03T05:24:20.835 DEBUG:teuthology.parallel:result is None 2024-06-03T05:24:20.892 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-03T05:24:21.509 INFO:teuthology.orchestra.run.smithi038.stdout:77309411339 2024-06-03T05:24:21.523 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411338 got 77309411339 for osd.4 2024-06-03T05:24:21.523 DEBUG:teuthology.parallel:result is None 2024-06-03T05:24:21.523 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-03T05:24:21.524 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:24:22.127 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:22.127 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:24:22.139 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-06-03T05:24:20.302549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218556,"kb_used_data":2360,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756164,"statfs":{"total":773094113280,"available":772870311936,"internally_reserved":0,"allocated":2416640,"data_stored":1352190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001236"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.824824+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:24:16.824824+0000","last_peered":"2024-06-03T05:24:16.824824+0000","last_clean":"2024-06-03T05:24:16.824824+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:24:16.824824+0000","last_undegraded":"2024-06-03T05:24:16.824824+0000","last_fullsized":"2024-06-03T05:24:16.824824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.754555+0000","last_change":"2024-06-03T05:24:15.538413+0000","last_active":"2024-06-03T05:24:16.754555+0000","last_peered":"2024-06-03T05:24:16.754555+0000","last_clean":"2024-06-03T05:24:16.754555+0000","last_became_active":"2024-06-03T05:24:15.511120+0000","last_became_peered":"2024-06-03T05:24:15.511120+0000","last_unstale":"2024-06-03T05:24:16.754555+0000","last_undegraded":"2024-06-03T05:24:16.754555+0000","last_fullsized":"2024-06-03T05:24:16.754555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:05:38.432196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.812350+0000","last_change":"2024-06-03T05:24:14.763971+0000","last_active":"2024-06-03T05:24:16.812350+0000","last_peered":"2024-06-03T05:24:16.812350+0000","last_clean":"2024-06-03T05:24:16.812350+0000","last_became_active":"2024-06-03T05:24:14.758379+0000","last_became_peered":"2024-06-03T05:24:14.758379+0000","last_unstale":"2024-06-03T05:24:16.812350+0000","last_undegraded":"2024-06-03T05:24:16.812350+0000","last_fullsized":"2024-06-03T05:24:16.812350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:01:23.148286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.688055+0000","last_change":"2024-06-03T05:24:15.223666+0000","last_active":"2024-06-03T05:24:16.688055+0000","last_peered":"2024-06-03T05:24:16.688055+0000","last_clean":"2024-06-03T05:24:16.688055+0000","last_became_active":"2024-06-03T05:24:15.216506+0000","last_became_peered":"2024-06-03T05:24:15.216506+0000","last_unstale":"2024-06-03T05:24:16.688055+0000","last_undegraded":"2024-06-03T05:24:16.688055+0000","last_fullsized":"2024-06-03T05:24:16.688055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:00:18.708313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.800031+0000","last_change":"2024-06-03T05:24:15.207972+0000","last_active":"2024-06-03T05:24:16.800031+0000","last_peered":"2024-06-03T05:24:16.800031+0000","last_clean":"2024-06-03T05:24:16.800031+0000","last_became_active":"2024-06-03T05:24:15.202389+0000","last_became_peered":"2024-06-03T05:24:15.202389+0000","last_unstale":"2024-06-03T05:24:16.800031+0000","last_undegraded":"2024-06-03T05:24:16.800031+0000","last_fullsized":"2024-06-03T05:24:16.800031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:55:27.456089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.627340+0000","last_change":"2024-06-03T05:24:14.681733+0000","last_active":"2024-06-03T05:24:16.627340+0000","last_peered":"2024-06-03T05:24:16.627340+0000","last_clean":"2024-06-03T05:24:16.627340+0000","last_became_active":"2024-06-03T05:24:14.652965+0000","last_became_peered":"2024-06-03T05:24:14.652965+0000","last_unstale":"2024-06-03T05:24:16.627340+0000","last_undegraded":"2024-06-03T05:24:16.627340+0000","last_fullsized":"2024-06-03T05:24:16.627340+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:38:49.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.768755+0000","last_change":"2024-06-03T05:24:14.688098+0000","last_active":"2024-06-03T05:24:16.768755+0000","last_peered":"2024-06-03T05:24:16.768755+0000","last_clean":"2024-06-03T05:24:16.768755+0000","last_became_active":"2024-06-03T05:24:14.659250+0000","last_became_peered":"2024-06-03T05:24:14.659250+0000","last_unstale":"2024-06-03T05:24:16.768755+0000","last_undegraded":"2024-06-03T05:24:16.768755+0000","last_fullsized":"2024-06-03T05:24:16.768755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:36:21.716172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.806334+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:24:16.806334+0000","last_peered":"2024-06-03T05:24:16.806334+0000","last_clean":"2024-06-03T05:24:16.806334+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:24:16.806334+0000","last_undegraded":"2024-06-03T05:24:16.806334+0000","last_fullsized":"2024-06-03T05:24:16.806334+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.670580+0000","last_change":"2024-06-03T05:24:12.154052+0000","last_active":"2024-06-03T05:24:16.670580+0000","last_peered":"2024-06-03T05:24:16.670580+0000","last_clean":"2024-06-03T05:24:16.670580+0000","last_became_active":"2024-06-03T05:24:11.479785+0000","last_became_peered":"2024-06-03T05:24:11.479785+0000","last_unstale":"2024-06-03T05:24:16.670580+0000","last_undegraded":"2024-06-03T05:24:16.670580+0000","last_fullsized":"2024-06-03T05:24:16.670580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:11:30.587750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378634,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411337,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411339,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":469097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":196608,"data_stored":66629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344244,"statfs":{"total":96636764160,"available":96608505856,"internally_reserved":0,"allocated":589824,"data_stored":460179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:24:22.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:24:22.747 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:22.748 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:24:22.761 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-06-03T05:24:22.302993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218584,"kb_used_data":2384,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756136,"statfs":{"total":773094113280,"available":772870283264,"internally_reserved":0,"allocated":2441216,"data_stored":1365736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.920998"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.824824+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:24:16.824824+0000","last_peered":"2024-06-03T05:24:16.824824+0000","last_clean":"2024-06-03T05:24:16.824824+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:24:16.824824+0000","last_undegraded":"2024-06-03T05:24:16.824824+0000","last_fullsized":"2024-06-03T05:24:16.824824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.754555+0000","last_change":"2024-06-03T05:24:15.538413+0000","last_active":"2024-06-03T05:24:16.754555+0000","last_peered":"2024-06-03T05:24:16.754555+0000","last_clean":"2024-06-03T05:24:16.754555+0000","last_became_active":"2024-06-03T05:24:15.511120+0000","last_became_peered":"2024-06-03T05:24:15.511120+0000","last_unstale":"2024-06-03T05:24:16.754555+0000","last_undegraded":"2024-06-03T05:24:16.754555+0000","last_fullsized":"2024-06-03T05:24:16.754555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:05:38.432196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.812350+0000","last_change":"2024-06-03T05:24:14.763971+0000","last_active":"2024-06-03T05:24:16.812350+0000","last_peered":"2024-06-03T05:24:16.812350+0000","last_clean":"2024-06-03T05:24:16.812350+0000","last_became_active":"2024-06-03T05:24:14.758379+0000","last_became_peered":"2024-06-03T05:24:14.758379+0000","last_unstale":"2024-06-03T05:24:16.812350+0000","last_undegraded":"2024-06-03T05:24:16.812350+0000","last_fullsized":"2024-06-03T05:24:16.812350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:01:23.148286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.688055+0000","last_change":"2024-06-03T05:24:15.223666+0000","last_active":"2024-06-03T05:24:16.688055+0000","last_peered":"2024-06-03T05:24:16.688055+0000","last_clean":"2024-06-03T05:24:16.688055+0000","last_became_active":"2024-06-03T05:24:15.216506+0000","last_became_peered":"2024-06-03T05:24:15.216506+0000","last_unstale":"2024-06-03T05:24:16.688055+0000","last_undegraded":"2024-06-03T05:24:16.688055+0000","last_fullsized":"2024-06-03T05:24:16.688055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:00:18.708313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.800031+0000","last_change":"2024-06-03T05:24:15.207972+0000","last_active":"2024-06-03T05:24:16.800031+0000","last_peered":"2024-06-03T05:24:16.800031+0000","last_clean":"2024-06-03T05:24:16.800031+0000","last_became_active":"2024-06-03T05:24:15.202389+0000","last_became_peered":"2024-06-03T05:24:15.202389+0000","last_unstale":"2024-06-03T05:24:16.800031+0000","last_undegraded":"2024-06-03T05:24:16.800031+0000","last_fullsized":"2024-06-03T05:24:16.800031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:55:27.456089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.627340+0000","last_change":"2024-06-03T05:24:14.681733+0000","last_active":"2024-06-03T05:24:16.627340+0000","last_peered":"2024-06-03T05:24:16.627340+0000","last_clean":"2024-06-03T05:24:16.627340+0000","last_became_active":"2024-06-03T05:24:14.652965+0000","last_became_peered":"2024-06-03T05:24:14.652965+0000","last_unstale":"2024-06-03T05:24:16.627340+0000","last_undegraded":"2024-06-03T05:24:16.627340+0000","last_fullsized":"2024-06-03T05:24:16.627340+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:38:49.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.768755+0000","last_change":"2024-06-03T05:24:14.688098+0000","last_active":"2024-06-03T05:24:16.768755+0000","last_peered":"2024-06-03T05:24:16.768755+0000","last_clean":"2024-06-03T05:24:16.768755+0000","last_became_active":"2024-06-03T05:24:14.659250+0000","last_became_peered":"2024-06-03T05:24:14.659250+0000","last_unstale":"2024-06-03T05:24:16.768755+0000","last_undegraded":"2024-06-03T05:24:16.768755+0000","last_fullsized":"2024-06-03T05:24:16.768755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:36:21.716172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.806334+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:24:16.806334+0000","last_peered":"2024-06-03T05:24:16.806334+0000","last_clean":"2024-06-03T05:24:16.806334+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:24:16.806334+0000","last_undegraded":"2024-06-03T05:24:16.806334+0000","last_fullsized":"2024-06-03T05:24:16.806334+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.670580+0000","last_change":"2024-06-03T05:24:12.154052+0000","last_active":"2024-06-03T05:24:16.670580+0000","last_peered":"2024-06-03T05:24:16.670580+0000","last_clean":"2024-06-03T05:24:16.670580+0000","last_became_active":"2024-06-03T05:24:11.479785+0000","last_became_peered":"2024-06-03T05:24:11.479785+0000","last_unstale":"2024-06-03T05:24:16.670580+0000","last_undegraded":"2024-06-03T05:24:16.670580+0000","last_fullsized":"2024-06-03T05:24:16.670580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:11:30.587750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378635,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411337,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411339,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":469097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":606208,"data_stored":469097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:24:22.762 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-03T05:24:22.762 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-03T05:24:22.762 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-06-03T05:24:22.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-06-03T05:24:23.405 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:23.405 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-03T05:24:23.419 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-06-03T05:24:23.420 INFO:teuthology.run_tasks:Running task thrashosds... 2024-06-03T05:24:23.430 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-06-03T05:24:23.430 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW OPS\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'dfcc4b3c2c55532c4c04fa47551c9df1dffdf746'}} 2024-06-03T05:24:23.430 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-06-03T05:24:23.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-06-03T05:24:24.229 INFO:tasks.thrashosds:Beginning thrashosds... 2024-06-03T05:24:24.229 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-03T05:24:24.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:24:24.833 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:24.834 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:24:24.846 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-06-03T05:24:24.303326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218584,"kb_used_data":2384,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756136,"statfs":{"total":773094113280,"available":772870283264,"internally_reserved":0,"allocated":2441216,"data_stored":1365736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.799140"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.824824+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:24:16.824824+0000","last_peered":"2024-06-03T05:24:16.824824+0000","last_clean":"2024-06-03T05:24:16.824824+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:24:16.824824+0000","last_undegraded":"2024-06-03T05:24:16.824824+0000","last_fullsized":"2024-06-03T05:24:16.824824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.754555+0000","last_change":"2024-06-03T05:24:15.538413+0000","last_active":"2024-06-03T05:24:16.754555+0000","last_peered":"2024-06-03T05:24:16.754555+0000","last_clean":"2024-06-03T05:24:16.754555+0000","last_became_active":"2024-06-03T05:24:15.511120+0000","last_became_peered":"2024-06-03T05:24:15.511120+0000","last_unstale":"2024-06-03T05:24:16.754555+0000","last_undegraded":"2024-06-03T05:24:16.754555+0000","last_fullsized":"2024-06-03T05:24:16.754555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:05:38.432196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.812350+0000","last_change":"2024-06-03T05:24:14.763971+0000","last_active":"2024-06-03T05:24:16.812350+0000","last_peered":"2024-06-03T05:24:16.812350+0000","last_clean":"2024-06-03T05:24:16.812350+0000","last_became_active":"2024-06-03T05:24:14.758379+0000","last_became_peered":"2024-06-03T05:24:14.758379+0000","last_unstale":"2024-06-03T05:24:16.812350+0000","last_undegraded":"2024-06-03T05:24:16.812350+0000","last_fullsized":"2024-06-03T05:24:16.812350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:01:23.148286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.688055+0000","last_change":"2024-06-03T05:24:15.223666+0000","last_active":"2024-06-03T05:24:16.688055+0000","last_peered":"2024-06-03T05:24:16.688055+0000","last_clean":"2024-06-03T05:24:16.688055+0000","last_became_active":"2024-06-03T05:24:15.216506+0000","last_became_peered":"2024-06-03T05:24:15.216506+0000","last_unstale":"2024-06-03T05:24:16.688055+0000","last_undegraded":"2024-06-03T05:24:16.688055+0000","last_fullsized":"2024-06-03T05:24:16.688055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:00:18.708313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.800031+0000","last_change":"2024-06-03T05:24:15.207972+0000","last_active":"2024-06-03T05:24:16.800031+0000","last_peered":"2024-06-03T05:24:16.800031+0000","last_clean":"2024-06-03T05:24:16.800031+0000","last_became_active":"2024-06-03T05:24:15.202389+0000","last_became_peered":"2024-06-03T05:24:15.202389+0000","last_unstale":"2024-06-03T05:24:16.800031+0000","last_undegraded":"2024-06-03T05:24:16.800031+0000","last_fullsized":"2024-06-03T05:24:16.800031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:55:27.456089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.627340+0000","last_change":"2024-06-03T05:24:14.681733+0000","last_active":"2024-06-03T05:24:16.627340+0000","last_peered":"2024-06-03T05:24:16.627340+0000","last_clean":"2024-06-03T05:24:16.627340+0000","last_became_active":"2024-06-03T05:24:14.652965+0000","last_became_peered":"2024-06-03T05:24:14.652965+0000","last_unstale":"2024-06-03T05:24:16.627340+0000","last_undegraded":"2024-06-03T05:24:16.627340+0000","last_fullsized":"2024-06-03T05:24:16.627340+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:38:49.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.768755+0000","last_change":"2024-06-03T05:24:14.688098+0000","last_active":"2024-06-03T05:24:16.768755+0000","last_peered":"2024-06-03T05:24:16.768755+0000","last_clean":"2024-06-03T05:24:16.768755+0000","last_became_active":"2024-06-03T05:24:14.659250+0000","last_became_peered":"2024-06-03T05:24:14.659250+0000","last_unstale":"2024-06-03T05:24:16.768755+0000","last_undegraded":"2024-06-03T05:24:16.768755+0000","last_fullsized":"2024-06-03T05:24:16.768755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:36:21.716172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.806334+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:24:16.806334+0000","last_peered":"2024-06-03T05:24:16.806334+0000","last_clean":"2024-06-03T05:24:16.806334+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:24:16.806334+0000","last_undegraded":"2024-06-03T05:24:16.806334+0000","last_fullsized":"2024-06-03T05:24:16.806334+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.670580+0000","last_change":"2024-06-03T05:24:12.154052+0000","last_active":"2024-06-03T05:24:16.670580+0000","last_peered":"2024-06-03T05:24:16.670580+0000","last_clean":"2024-06-03T05:24:16.670580+0000","last_became_active":"2024-06-03T05:24:11.479785+0000","last_became_peered":"2024-06-03T05:24:11.479785+0000","last_unstale":"2024-06-03T05:24:16.670580+0000","last_undegraded":"2024-06-03T05:24:16.670580+0000","last_fullsized":"2024-06-03T05:24:16.670580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:11:30.587750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378635,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411337,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411339,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":469097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":606208,"data_stored":469097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:24:24.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:24:25.467 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:25.467 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:24:25.479 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-06-03T05:24:24.303326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218584,"kb_used_data":2384,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756136,"statfs":{"total":773094113280,"available":772870283264,"internally_reserved":0,"allocated":2441216,"data_stored":1365736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.799140"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.824824+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:24:16.824824+0000","last_peered":"2024-06-03T05:24:16.824824+0000","last_clean":"2024-06-03T05:24:16.824824+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:24:16.824824+0000","last_undegraded":"2024-06-03T05:24:16.824824+0000","last_fullsized":"2024-06-03T05:24:16.824824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.754555+0000","last_change":"2024-06-03T05:24:15.538413+0000","last_active":"2024-06-03T05:24:16.754555+0000","last_peered":"2024-06-03T05:24:16.754555+0000","last_clean":"2024-06-03T05:24:16.754555+0000","last_became_active":"2024-06-03T05:24:15.511120+0000","last_became_peered":"2024-06-03T05:24:15.511120+0000","last_unstale":"2024-06-03T05:24:16.754555+0000","last_undegraded":"2024-06-03T05:24:16.754555+0000","last_fullsized":"2024-06-03T05:24:16.754555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:05:38.432196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.812350+0000","last_change":"2024-06-03T05:24:14.763971+0000","last_active":"2024-06-03T05:24:16.812350+0000","last_peered":"2024-06-03T05:24:16.812350+0000","last_clean":"2024-06-03T05:24:16.812350+0000","last_became_active":"2024-06-03T05:24:14.758379+0000","last_became_peered":"2024-06-03T05:24:14.758379+0000","last_unstale":"2024-06-03T05:24:16.812350+0000","last_undegraded":"2024-06-03T05:24:16.812350+0000","last_fullsized":"2024-06-03T05:24:16.812350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:01:23.148286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.688055+0000","last_change":"2024-06-03T05:24:15.223666+0000","last_active":"2024-06-03T05:24:16.688055+0000","last_peered":"2024-06-03T05:24:16.688055+0000","last_clean":"2024-06-03T05:24:16.688055+0000","last_became_active":"2024-06-03T05:24:15.216506+0000","last_became_peered":"2024-06-03T05:24:15.216506+0000","last_unstale":"2024-06-03T05:24:16.688055+0000","last_undegraded":"2024-06-03T05:24:16.688055+0000","last_fullsized":"2024-06-03T05:24:16.688055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:00:18.708313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.800031+0000","last_change":"2024-06-03T05:24:15.207972+0000","last_active":"2024-06-03T05:24:16.800031+0000","last_peered":"2024-06-03T05:24:16.800031+0000","last_clean":"2024-06-03T05:24:16.800031+0000","last_became_active":"2024-06-03T05:24:15.202389+0000","last_became_peered":"2024-06-03T05:24:15.202389+0000","last_unstale":"2024-06-03T05:24:16.800031+0000","last_undegraded":"2024-06-03T05:24:16.800031+0000","last_fullsized":"2024-06-03T05:24:16.800031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:55:27.456089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.627340+0000","last_change":"2024-06-03T05:24:14.681733+0000","last_active":"2024-06-03T05:24:16.627340+0000","last_peered":"2024-06-03T05:24:16.627340+0000","last_clean":"2024-06-03T05:24:16.627340+0000","last_became_active":"2024-06-03T05:24:14.652965+0000","last_became_peered":"2024-06-03T05:24:14.652965+0000","last_unstale":"2024-06-03T05:24:16.627340+0000","last_undegraded":"2024-06-03T05:24:16.627340+0000","last_fullsized":"2024-06-03T05:24:16.627340+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:38:49.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.768755+0000","last_change":"2024-06-03T05:24:14.688098+0000","last_active":"2024-06-03T05:24:16.768755+0000","last_peered":"2024-06-03T05:24:16.768755+0000","last_clean":"2024-06-03T05:24:16.768755+0000","last_became_active":"2024-06-03T05:24:14.659250+0000","last_became_peered":"2024-06-03T05:24:14.659250+0000","last_unstale":"2024-06-03T05:24:16.768755+0000","last_undegraded":"2024-06-03T05:24:16.768755+0000","last_fullsized":"2024-06-03T05:24:16.768755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:36:21.716172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.806334+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:24:16.806334+0000","last_peered":"2024-06-03T05:24:16.806334+0000","last_clean":"2024-06-03T05:24:16.806334+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:24:16.806334+0000","last_undegraded":"2024-06-03T05:24:16.806334+0000","last_fullsized":"2024-06-03T05:24:16.806334+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-06-03T05:24:16.670580+0000","last_change":"2024-06-03T05:24:12.154052+0000","last_active":"2024-06-03T05:24:16.670580+0000","last_peered":"2024-06-03T05:24:16.670580+0000","last_clean":"2024-06-03T05:24:16.670580+0000","last_became_active":"2024-06-03T05:24:11.479785+0000","last_became_peered":"2024-06-03T05:24:11.479785+0000","last_unstale":"2024-06-03T05:24:16.670580+0000","last_undegraded":"2024-06-03T05:24:16.670580+0000","last_fullsized":"2024-06-03T05:24:16.670580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:11:30.587750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378635,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411337,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411339,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":469097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":71257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":606208,"data_stored":469097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:24:25.480 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-03T05:24:25.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-06-03T05:24:26.103 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 24 2024-06-03T05:24:26.103 INFO:teuthology.orchestra.run.smithi038.stdout:fsid ebc09f7e-f094-4421-826f-479fc526fc62 2024-06-03T05:24:26.103 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-06-03T05:23:10.198378+0000 2024-06-03T05:24:26.103 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-06-03T05:24:16.465229+0000 2024-06-03T05:24:26.103 INFO:teuthology.orchestra.run.smithi038.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-06-03T05:24:26.103 INFO:teuthology.orchestra.run.smithi038.stdout:crush_version 6 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:full_ratio 0.95 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:backfillfull_ratio 0.9 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:nearfull_ratio 0.85 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:require_min_compat_client luminous 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:min_compat_client jewel 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:require_osd_release squid 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:stretch_mode_enabled false 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 24 flags hashpspool stripe_width 0 application rbd read_balance_score 4.98 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:max_osd 8 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6809/905141127 v2:172.21.15.38:6811/905141127 exists,up 32b256d9-eee7-435b-8aec-fef2ba90366c 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 up in weight 1 up_from 18 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6810/3490615740 v2:172.21.15.38:6812/3490615740 exists,up 371c2e00-54db-430b-8497-a4c9ff6753b9 2024-06-03T05:24:26.104 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6805/3359326199 v2:172.21.15.38:6806/3359326199 exists,up e8145335-ec5d-4dd1-9250-a4ab5b159d29 2024-06-03T05:24:26.105 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 up in weight 1 up_from 19 up_thru 21 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6801/2289724788 v2:172.21.15.38:6802/2289724788 exists,up 1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c 2024-06-03T05:24:26.105 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6808/3833397294 v2:172.21.15.64:6809/3833397294 exists,up 46b46f54-04d0-46c0-809c-64fa385e6780 2024-06-03T05:24:26.105 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 up in weight 1 up_from 13 up_thru 21 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6800/3915827095 v2:172.21.15.64:6801/3915827095 exists,up f7a50ac0-eba9-4602-b90a-2dbc4bb20593 2024-06-03T05:24:26.105 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6804/1144336651 v2:172.21.15.64:6805/1144336651 exists,up 081024c0-2482-4846-a7bd-594a58f2ceef 2024-06-03T05:24:26.105 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6812/370645108 v2:172.21.15.64:6813/370645108 exists,up 6234c757-8f7b-497e-9347-4a7983df592c 2024-06-03T05:24:26.117 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6809/905141127 v2:172.21.15.38:6811/905141127 exists,up 32b256d9-eee7-435b-8aec-fef2ba90366c', 'osd.1 up in weight 1 up_from 18 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6810/3490615740 v2:172.21.15.38:6812/3490615740 exists,up 371c2e00-54db-430b-8497-a4c9ff6753b9', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6805/3359326199 v2:172.21.15.38:6806/3359326199 exists,up e8145335-ec5d-4dd1-9250-a4ab5b159d29', 'osd.3 up in weight 1 up_from 19 up_thru 21 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6801/2289724788 v2:172.21.15.38:6802/2289724788 exists,up 1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c', 'osd.4 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6808/3833397294 v2:172.21.15.64:6809/3833397294 exists,up 46b46f54-04d0-46c0-809c-64fa385e6780', 'osd.5 up in weight 1 up_from 13 up_thru 21 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6800/3915827095 v2:172.21.15.64:6801/3915827095 exists,up f7a50ac0-eba9-4602-b90a-2dbc4bb20593', 'osd.6 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6804/1144336651 v2:172.21.15.64:6805/1144336651 exists,up 081024c0-2482-4846-a7bd-594a58f2ceef', 'osd.7 up in weight 1 up_from 18 up_thru 21 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6812/370645108 v2:172.21.15.64:6813/370645108 exists,up 6234c757-8f7b-497e-9347-4a7983df592c'] 2024-06-03T05:24:26.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mon.a", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster": "ceph", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "admin_socket_mode": "", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_client_required": "cephx, none", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_cluster_required": "cephx", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_debug": "true", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-06-03T05:24:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_service_required": "cephx", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_service_ticket_ttl": "240.000000", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_supported": "cephx", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio": "true", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_max_queue_depth": "1024", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_poll_ms": "250", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_reap_max": "16", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_async_discard": "false", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_block_size": "4096", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio": "false", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-06-03T05:24:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_inflight_ios": "false", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_enable_discard": "false", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_flock_retry": "3", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_flock_retry_interval": "0.100000", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_inject_crash": "0", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_inject_crash_flush_delay": "2", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring": "false", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring_hipri": "false", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring_sqthread_poll": "false", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_read_buffer_alignment": "4096", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_type": "", 2024-06-03T05:24:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_alloc_size": "1048576", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_allocator": "hybrid", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_buffered_io": "true", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_for_zeros": "false", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_volume_selector_often": "false", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_compact_log_sync": "false", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_compact_min_size": "16777216", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_replay_check_allocations": "true", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_max_log_runway": "4194304", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_max_prefetch": "1048576", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_min_flush_size": "524288", 2024-06-03T05:24:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_min_log_runway": "1048576", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_replay_recovery": "false", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_shared_alloc_size": "65536", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_sync_write": "false", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_allocation_from_file": "true", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_allocator": "hybrid", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-06-03T05:24:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blobid_prealloc": "10240", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_create": "true", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_create": "false", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_path": "", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_size": "0", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_path": "", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_preallocate_file": "false", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_size": "107374182400", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_create": "false", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_path": "", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_size": "100663296", 2024-06-03T05:24:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs": "true", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_env_mirror": "false", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_autotune": "true", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-06-03T05:24:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size": "0", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_type": "2q", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_clone_cow": "true", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_algorithm": "snappy", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size": "0", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size": "0", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-06-03T05:24:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_mode": "none", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_csum_type": "crc32c", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_enforce_settings": "default", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_freelist": "false", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_fsck_abort": "false", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_read_err": "false", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_legacy_omap": "false", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-06-03T05:24:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_prefill": "0.000000", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_small_allocations": "0", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_default_buffered_read": "true", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_default_buffered_write": "false", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops": "0", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_elastic_shared_blobs": "true", 2024-06-03T05:24:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fail_eio": "false", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mkfs": "true", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mount": "false", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-06-03T05:24:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_umount": "false", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_ignore_data_csum": "false", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_kvbackend": "rocksdb", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-06-03T05:24:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_op_age": "5.000000", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_alloc_size": "0", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size": "0", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_defer_interval": "3.000000", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_deferred_txc": "32", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size": "0", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_nid_prealloc": "1024", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size": "0", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-06-03T05:24:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_qfsck_on_mount": "true", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_retry_disk_reads": "3", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_cf": "true", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_options_annex": "", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_coremask": "0x1", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_io_sleep": "5", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_max_io_completion": "0", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_mem": "512", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_sync_submit_transaction": "false", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_bytes": "67108864", 2024-06-03T05:24:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io": "0", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_tracing": "false", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_reserved": "0", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-06-03T05:24:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_zero_block_detection": "false", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_action_update_interval": "2", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_mount_timeout": "10", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-06-03T05:24:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_cluster_require_signatures": "false", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_cluster_require_version": "2", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_require_signatures": "false", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_require_version": "2", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_service_require_signatures": "false", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_service_require_version": "2", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_sign_messages": "true", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "chdir": "", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_acl_type": "", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_asio_thread_count": "2", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_cache_mid": "0.750000", 2024-06-03T05:24:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_cache_size": "16384", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_caps_release_delay": "5", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_check_pool_perm": "true", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_collect_and_send_global_metrics": "false", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_force_sync_read": "false", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_getattr_caps": "false", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_inject_features": "", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_inject_tick_delay": "0", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_die_on_failed_remount": "false", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_dirsize_rbytes": "true", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_force_lazyio": "false", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_fs": "", 2024-06-03T05:24:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_inject_fixed_oldest_tid": "false", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_inject_release_failure": "false", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_max_inline_size": "4096", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_max_retries_on_remount_failure": "5", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mds_namespace": "", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_metadata": "", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_gid": "-1", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_timeout": "300", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_uid": "-1", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mountpoint": "/", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_notify_timeout": "10", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc": "true", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_dirty": "104857600", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_dirty_age": "5.000000", 2024-06-03T05:24:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_objects": "1000", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_size": "209715200", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_target_dirty": "8388608", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_permissions": "true", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_quota": "true", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_quota_df": "true", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_max_bytes": "0", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_max_periods": "4", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_min": "131072", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_reconnect_stale": "false", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_shutdown_timeout": "30", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_snapdir": ".snap", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_tick_interval": "1", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_trace": "", 2024-06-03T05:24:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_try_dentry_invalidate": "false", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client_use_faked_inos": "false", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client_use_random_mds": "false", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog": "false", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog_port": "12201", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_monitors": "default=true", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog": "false", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog_level": "info", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "-", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_network": "", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_network_interface": "", 2024-06-03T05:24:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_isal": "false", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_level": "5", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_winsize": "-15", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zstd_level": "1", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_alien_op_num_threads": "6", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_alien_thread_cpu_cores": "", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_obc_lru_size": "512", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_stat_interval": "0", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_seastar_cpu_cores": "", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "", 2024-06-03T05:24:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location_hook": "", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location_hook_timeout": "10", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "daemonize": "false", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "daos_pool": "tank", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_db_name_prefix": "dbstore", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_alienstore": "0/5", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_allow_any_pool_priority": "false", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asok": "1/5", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asok_assert_abort": "false", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asserts_on_shutdown": "true", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_auth": "20/20", 2024-06-03T05:24:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bdev": "1/3", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bluefs": "1/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bluestore": "1/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_buffer": "0/1", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_ceph_exporter": "1/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cephfs_mirror": "0/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cephsqlite": "0/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_client": "0/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_compressor": "1/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_context": "0/1", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crimson_interrupt": "0/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crush": "1/1", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crypto": "1/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cyanstore": "0/5", 2024-06-03T05:24:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_deliberately_leak_memory": "false", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_disable_randomized_ping": "false", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_dpdk": "1/5", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_eventtrace": "1/5", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_filer": "0/1", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_filestore": "1/3", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_finisher": "1/1", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_fuse": "1/5", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_heartbeat_testing_span": "0", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_heartbeatmap": "1/5", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_immutable_obj_cache": "0/5", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_javaclient": "1/5", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_journal": "1/3", 2024-06-03T05:24:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_journaler": "0/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_kstore": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_lockdep": "0/1", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mclock": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_balancer": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_locker": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_log": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_log_expire": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_migrator": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_quiesce": "3/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_memstore": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mgr": "2/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mgrc": "1/5", 2024-06-03T05:24:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mon": "20/20", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_monc": "0/10", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_ms": "1/1", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_none": "0/5", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objclass": "0/5", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objectcacher": "0/5", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objecter": "0/1", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_optracker": "0/5", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_osd": "1/5", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_paxos": "20/20", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_perfcounter": "1/5", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_prioritycache": "1/5", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rados": "0/5", 2024-06-03T05:24:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd": "0/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_mirror": "0/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_pwl": "0/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_replay": "0/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_refs": "0/0", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_reserver": "1/1", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw": "1/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_access": "1/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_datacache": "1/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_dbstore": "1/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_flight": "1/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_lifecycle": "1/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_sync": "1/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rocksdb": "4/5", 2024-06-03T05:24:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_backref": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_cache": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_cleaner": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_device": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_epm": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_journal": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_lba": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_odata": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_omap": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_onode": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_t": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_tm": "0/5", 2024-06-03T05:24:26.333 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_striper": "0/1", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_test": "0/5", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_throttle": "1/1", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_timer": "0/1", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_tp": "0/5", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_trace": "1/5", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "device_failure_prediction_mode": "none", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_graylog": "false", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_journald": "false", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_stderr": "true", 2024-06-03T05:24:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_syslog": "false", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event_tracing": "false", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_addr": "0.0.0.0", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_http_port": "9926", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_prio_limit": "5", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_sort_metrics": "true", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_stats_period": "5", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "fake_statfs_for_testing": "0", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "fatal_signal_handlers": "true", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "filer_max_purge_ops": "10", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "filer_max_truncate_ops": "128", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_apply_finisher_threads": "1", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_blackhole": "false", 2024-06-03T05:24:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_btrfs_clone_range": "true", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_btrfs_snap": "true", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_caller_concurrency": "10", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_collect_device_partition_information": "true", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_commit_timeout": "600.000000", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_inject_read_err": "false", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_omap_check": "false", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_random_read_err": "0.000000", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_verify_split": "false", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_dump_file": "", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fadvise": "true", 2024-06-03T05:24:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fail_eio": "true", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fd_cache_shards": "16", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fd_cache_size": "128", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fiemap": "false", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fiemap_threshold": "4096", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_index_retry_probability": "0.000000", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_inject_stall": "0", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_parallel": "false", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_trailing": "false", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_writeahead": "false", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_kill_at": "0", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-06-03T05:24:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size": "0", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs": "0", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_other": "2", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_sync_interval": "5.000000", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size": "0", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-06-03T05:24:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_merge_threshold": "-10", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_min_sync_interval": "0.010000", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_odsync_write": "false", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_backend": "rocksdb", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_backend_path": "", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_header_cache_size": "1024", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_ondisk_finisher_threads": "1", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_thread_timeout": "60", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_threads": "2", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_punch_hole": "false", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-06-03T05:24:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_bytes": "104857600", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_ops": "50", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_seek_data_hole": "false", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_sloppy_crc": "false", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_splice": "false", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_split_multiple": "2", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_split_rand_factor": "20", 2024-06-03T05:24:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_update_to": "1000", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_enable": "true", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-06-03T05:24:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_xfs_extsize": "false", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_zfs_snap": "false", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fio_dir": "/tmp/fio", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "ebc09f7e-f094-4421-826f-479fc526fc62", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_allow_other": "true", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_atomic_o_trunc": "true", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_big_writes": "true", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_debug": "false", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_default_permissions": "false", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_disable_pagecache": "false", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_max_write": "0", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_multithreaded": "true", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_require_active_mds": "true", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_set_user_groups": "true", 2024-06-03T05:24:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_move": "true", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_read": "true", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_write": "true", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_syncfs_on_mksnap": "true", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_use_invalidate_cb": "true", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "gss_target_name": "ceph", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_file": "", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_inject_failure": "0", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_interval": "5", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "host": "smithi038", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_max_size": "1073741824", 2024-06-03T05:24:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_watermark": "0.900000", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "inject_early_sigterm": "false", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "jaeger_agent_port": "6799", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "jaeger_tracing_enable": "false", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_aio": "true", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_align_min_size": "65536", 2024-06-03T05:24:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_block_align": "true", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_block_size": "4096", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_dio": "true", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_discard": "false", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_force_aio": "false", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_ignore_corruption": "false", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_max_write_bytes": "10485760", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_max_write_entries": "100", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_replay_from": "0", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_high_multiple": "0.000000", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_max_multiple": "0.000000", 2024-06-03T05:24:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_write_header_frequency": "0", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_zero_on_create": "false", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_prefetch_periods": "10", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_prezero_periods": "5", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_write_head_interval": "15", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "keyfile": "", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_backend": "rocksdb", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_default_stripe_size": "65536", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_fsck_on_mount": "false", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_fsck_on_mount_deep": "true", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_max_bytes": "67108864", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_max_ops": "512", 2024-06-03T05:24:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_nid_prealloc": "1024", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_onode_map_size": "1024", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_sync_submit_transaction": "false", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_sync_transaction": "false", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "librados_thread_count": "2", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "lockdep": "false", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "lockdep_force_backtrace": "false", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_coarse_timestamps": "true", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_flush_on_exit": "false", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_graylog_host": "127.0.0.1", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_graylog_port": "12201", 2024-06-03T05:24:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_max_new": "1000", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_max_recent": "10000", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_stderr_prefix": "", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_stop_at_utilization": "0.970000", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": "true", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_graylog": "false", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_journald": "false", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_stderr": "false", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_syslog": "false", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "max_rotating_auth_attempts": "10", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_action_on_write_error": "1", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_alternate_name_max": "8192", 2024-06-03T05:24:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_asio_thread_count": "2", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_export_pin": "true", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_dirs": "true", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_interval": "5", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_size_max": "100000", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_idle_threshold": "0.000000", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_interval": "10", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_max": "-1", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_max_until": "-1", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_merge_size": "50", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_midchunk": "0.300000", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_min_rebalance": "0.100000", 2024-06-03T05:24:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_min_start": "0.200000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_minchunk": "0.001000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_mode": "0", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_need_max": "1.200000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_need_min": "0.800000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_overload_epochs": "2", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_sample_interval": "3.000000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_bits": "3", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_rd": "25000.000000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_size": "10000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_wr": "10000.000000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_target_decay": "10.000000", 2024-06-03T05:24:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_grace": "15.000000", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_interval": "4.000000", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_mon_down_grace": "60", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_memory_limit": "4294967296", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_mid": "0.700000", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_delay": "0", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_sleep": "200", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_splitauth": "true", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_threshold": "524288", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_release_free_interval": "10", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_reservation": "0.050000", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-06-03T05:24:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_interval": "1", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_threshold": "262144", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_delegate_inos_pct": "50", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_prealloc_inos": "1000", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_connect_bootstrapping": "false", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_damage_table_max_entries": "10000", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_auth_pins": "false", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_frag": "false", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_scatterstat": "false", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_subtrees": "false", 2024-06-03T05:24:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_decay_halflife": "5.000000", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_default_dir_hash": "2", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_defer_session_stale": "true", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_deny_all_reconnect": "false", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_keys_per_op": "16384", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_max_commit_size": "10", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_max_entries": "0", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_prefetch": "true", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dirstat_min_interval": "1.000000", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_after_rejoin": "false", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_on_map": "false", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_threshold_file": "0", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_early_reply": "true", 2024-06-03T05:24:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_enable_op_tracker": "true", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_enforce_unique_name": "true", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_distributed": "true", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_random": "true", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_forward_all_requests_to_auth": "false", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_health_cache_threshold": "1.500000", 2024-06-03T05:24:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_health_summarize_threshold": "10", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_heartbeat_grace": "15.000000", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_heartbeat_reset_grace": "1000", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_health_dummy": "false", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_migrator_session_race": "false", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_join_fs": "", 2024-06-03T05:24:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_journal_format": "1", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_create_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_export_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_import_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_expire_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_replay_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_link_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_mdstable_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_openc_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_rename_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_shutdown_at": "0", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_event_large_threshold": "524288", 2024-06-03T05:24:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_events_per_segment": "1024", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_major_segment_event_ratio": "12", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_max_events": "-1", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_max_segments": "128", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_pause": "false", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_segment_size": "0", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_skip_corrupt_events": "false", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_skip_unbounded_events": "false", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_threshold": "128", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_warn_factor": "2.000000", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_caps_per_client": "1048576", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_completed_flushes": "100000", 2024-06-03T05:24:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_completed_requests": "100000", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_export_size": "20971520", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_file_recover": "32", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_files": "64", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_ops": "8192", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_snaps_per_dir": "100", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_metrics_update_interval": "2", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_min_caps_per_client": "100", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_min_caps_working_set": "10000", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_numa_node": "-1", 2024-06-03T05:24:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_complaint_time": "30.000000", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_duration": "600", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_size": "20", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_slow_op_size": "20", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_slow_op_threshold": "10", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_log_threshold": "5", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_ping_grace": "15", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_ping_interval": "5", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_caps": "30000", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_decay_threshold": "131072", 2024-06-03T05:24:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_warning_threshold": "262144", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_reconnect_timeout": "45.000000", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_replay_interval": "1.000000", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_root_ino_gid": "0", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_root_ino_uid": "0", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_blocklist_on_evict": "true", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_blocklist_on_timeout": "true", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-06-03T05:24:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-06-03T05:24:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-06-03T05:24:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_metadata_threshold": "16777216", 2024-06-03T05:24:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-06-03T05:24:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_shutdown_check": "0", 2024-06-03T05:24:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_skip_ino": "0", 2024-06-03T05:24:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sleep_rank_change": "0.000000", 2024-06-03T05:24:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_max_uid": "4294967294", 2024-06-03T05:24:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_min_uid": "0", 2024-06-03T05:24:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_rstat": "false", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_standby_replay_damaged": "false", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_symlink_recovery": "true", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_task_status_update_interval": "2.000000", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_thrash_exports": "0", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_thrash_fragments": "0", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_tick_interval": "5.000000", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_valgrind_exit": "false", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_verify_backtrace": "1", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_verify_scatter": "false", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_wipe_ino_prealloc": "false", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_wipe_sessions": "false", 2024-06-03T05:24:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool_debug": "false", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_debug_omit_block_device_write": "false", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_device_bytes": "1073741824", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_page_set": "false", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_page_size": "65536", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_bytes": "134217728", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_messages": "512", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_connect_retry_interval": "1.000000", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_disabled_modules": "", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-06-03T05:24:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_max_pg_num_change": "128", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mds_bytes": "134217728", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mds_messages": "128", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mon_bytes": "134217728", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mon_messages": "128", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_osd_bytes": "536870912", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_osd_messages": "8192", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_pool": "true", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_service_beacon_grace": "60.000000", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_standby_modules": "true", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_stats_period": "5", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_stats_threshold": "5", 2024-06-03T05:24:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_tick_period": "2", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_accept_timeout_factor": "2.000000", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_allow_pool_delete": "true", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_allow_pool_size_one": "true", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_auth_validate_all_caps": "true", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_bytes": "104857600", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_directed_command_retry": "5", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval": "3.000000", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-06-03T05:24:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_parallel": "3", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_log_interval": "1.000000", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_ping_interval": "10.000000", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_ping_timeout": "30.000000", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_target_rank": "-1", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clock_drift_allowed": "1.000000", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_file_level": "debug", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_file": "true", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog": "false", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-06-03T05:24:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_journald": "false", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_stderr": "false", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_bootstrap": "false", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_start": "false", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_trim": "true", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_con_tracker_persist_interval": "10", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_con_tracker_score_halflife": "43200", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_config_key_max_entry_size": "65536", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cpu_threads": "4", 2024-06-03T05:24:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_crush_min_required_version": "hammer", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_daemon_bytes": "419430400", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_avail_crit": "5", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_avail_warn": "5", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_size_warn": "16106127360", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_block_osdmap_trim": "false", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_json": "false", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_transactions": "false", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_extra_checks": "false", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-06-03T05:24:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_reef": "false", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_squid": "false", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_delta_reset_interval": "10.000000", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_dns_srv_name": "ceph-mon", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_down_mkfs_grace": "120", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_down_uptime_grace": "60", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_election_default_strategy": "1", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_election_timeout": "5.000000", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ping_divisor": "2", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ping_timeout": "2.000000", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_enable_op_tracker": "true", 2024-06-03T05:24:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_fake_pool_delete": "false", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_force_quorum_join": "false", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_globalid_prealloc": "10000", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_detail_to_clog": "false", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_log_update_period": "5", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_max_detail": "50", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog": "true", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog_interval": "600", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_host": "172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.64", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_host_override": "", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_initial_members": "", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-06-03T05:24:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_keyvaluedb": "rocksdb", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease": "5.000000", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_full_interval": "50", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_max": "10000", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_max_summary": "50", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_log_entries_per_event": "4096", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_log_epochs": "500", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_mdsmap_epochs": "500", 2024-06-03T05:24:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_mgrmap_epochs": "500", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_osd": "10000", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_pg_per_osd": "10000", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_pool_pg_num": "65536", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_force_trim_to": "0", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_skip_sanity": "false", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_memory_autotune": "true", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_memory_target": "2147483648", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_beacon_grace": "30", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_digest_period": "5", 2024-06-03T05:24:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_inactive_grace": "60", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_mkfs_grace": "240", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_min_osdmap_epochs": "50", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_complaint_time": "30", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_duration": "600", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_size": "20", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_slow_op_size": "20", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_slow_op_threshold": "10", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_log_threshold": "5", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_in": "false", 2024-06-03T05:24:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_new_in": "true", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_cache_size": "500", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_cache_size_min": "134217728", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_crush_smoke_test": "false", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_destroyed_out_interval": "600", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_down_out_interval": "600", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_force_trim_to": "0", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_full_ratio": "0.950000", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-06-03T05:24:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_halflife": "3600", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_max_interval": "300", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_weight": "0.300000", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_max_creating_pgs": "1024", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_max_initial_pgs": "1024", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_down_reporters": "2", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp": "true", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-06-03T05:24:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_report_timeout": "900", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_warn_num_repaired": "10", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_warn_op_age": "32.000000", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_interval": "2", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_min": "15", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_stuck_threshold": "60", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_objects": "10000", 2024-06-03T05:24:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_per_osd": "0", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pool_quota_crit_threshold": "0", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pool_quota_warn_threshold": "0", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_probe_timeout": "2.000000", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_max_change": "0.050000", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_max_osds": "4", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_interval": "300", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_max_keys": "100", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_timeout": "300", 2024-06-03T05:24:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_session_timeout": "300", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_smart_report_timeout": "5", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stat_smooth_intervals": "6", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_pool_min_size": "2", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_pool_size": "4", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_subscribe_interval": "86400.000000", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_debug": "false", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_max_payload_keys": "2000", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_max_payload_size": "1048576", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_provider_kill_at": "0", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_requester_kill_at": "0", 2024-06-03T05:24:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_timeout": "60.000000", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_target_pg_per_osd": "100", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_tick_interval": "5", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_timecheck_interval": "300.000000", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_use_min_delay_socket": "false", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_older_version_delay": "604800", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_filestore_osds": "false", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-06-03T05:24:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_misplaced": "false", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_older_version": "true", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_no_app": "true", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_too_few_osds": "false", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-06-03T05:24:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "monmap": "", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_tracing_enabled": "false", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_op_threads": "3", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_buffer_size": "131072", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_cm": "false", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_device_name": "", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_dscp": "96", 2024-06-03T05:24:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_gid_idx": "0", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_local_gid": "", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_polling_us": "1000", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_port_num": "1", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_roce_ver": "1", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_send_buffers": "1024", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_sl": "3", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_support_srq": "true", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_type": "ib", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_reap_threshold": "5", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_before_connect": "false", 2024-06-03T05:24:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_ipv4": "true", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_ipv6": "false", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_msgr1": "false", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_msgr2": "true", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_port_max": "7568", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_port_min": "6800", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_prefer_ipv4": "false", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_retry_count": "3", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_retry_delay": "5", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_client": "false", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mds": "false", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mgr": "false", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mon": "false", 2024-06-03T05:24:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_osd": "false", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_client_mode": "crc secure", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_cluster_mode": "crc secure", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_cluster_type": "", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_compress_secure": "false", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_connection_idle_timeout": "900", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_connection_ready_timeout": "10", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_crc_data": "true", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_crc_header": "true", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_bad_msg": "false", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_bug": "true", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_old_message": "true", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_skipped_message": "false", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_unhandled_msg": "false", 2024-06-03T05:24:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_coremask": "0xF", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_devs_allowlist": "", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_enable_tso": "true", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hugepages": "", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hw_flow_control": "true", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_lro": "true", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_memory_channel": "4", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_pmd": "", 2024-06-03T05:24:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_port_id": "0", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dump_corrupt_message_level": "1", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dump_on_send": "false", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_initial_backoff": "0.200000", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_max": "1.000000", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_probability": "0.000000", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_type": "", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_internal_delays": "0.000000", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_network_congestion": "0", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_socket_failures": "5000", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_learn_addr_from_peer": "true", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_max_accept_failures": "4", 2024-06-03T05:24:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_max_backoff": "15.000000", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_client_mode": "secure crc", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_cluster_mode": "secure crc", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_service_mode": "secure crc", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compress_min_size": "1024", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compress_mode": "none", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compression_algorithm": "snappy", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_pq_min_cost": "65536", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_public_type": "", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_service_mode": "crc secure", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_listen_backlog": "512", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_nodelay": "true", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-06-03T05:24:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_rcvbuf": "0", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_type": "async", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "no_config_file": "false", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_completion_locks_per_session": "32", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_debug_inject_relock_delay": "false", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inflight_op_bytes": "104857600", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inflight_ops": "1024", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inject_no_watch_ping": "false", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_tick_interval": "5.000000", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_timeout": "10.000000", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objectstore_blackhole": "false", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "openssl_engine_opts": "", 2024-06-03T05:24:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_delay_time": "5.000000", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_hist_halflife": "1000", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_max_low_ops": "2", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_max_ops": "4", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_quantize_effort": "0.100000", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_slop": "0.020000", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_allow_recovery_below_min_size": "true", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_asio_thread_count": "2", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_async_recovery_min_cost": "100", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_auto_mark_unfound_lost": "false", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_retry_interval": "30.000000", 2024-06-03T05:24:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_scan_max": "512", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_scan_min": "64", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_degraded": "false", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_peering": "false", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_unfound": "true", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_beacon_report_interval": "300", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_duration": "30", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_max_block_size": "67108864", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_small_size_max_iops": "100", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_blkin_trace_all": "false", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_blocked_scrub_grace_period": "120", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-06-03T05:24:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_check_for_log_corruption": "false", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_update_on_start": "true", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_message_cap": "256", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_message_size_cap": "524288000", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_op_priority": "63", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_watch_timeout": "30", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_command_max_records": "256", 2024-06-03T05:24:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_compact_on_start": "false", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_chooseleaf_type": "0", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_initial_weight": "-1.000000", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_update_on_start": "true", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_update_weight_set": "true", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_drop_ping_duration": "0", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_feed_pullee": "-1", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-06-03T05:24:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_misdirected_ops": "false", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_no_acting_change": "false", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_no_purge_strays": "false", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_op_order": "false", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_pg_log_writeout": "false", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_pretend_recovery_active": "false", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_shutdown": "false", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-06-03T05:24:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_trim_objects": "false", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_cached_snaps": "false", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_missing_on_start": "false", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_snaps": "false", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_keys": "1024", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_stride": "524288", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_default_data_pool_replay_window": "5", 2024-06-03T05:24:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_default_notify_timeout": "30", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep": "0.000000", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_discard_disconnected_ops": "true", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_enable_op_tracker": "true", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_extblkdev_plugins": "vdo", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_info": "true", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown": "true", 2024-06-03T05:24:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown_timeout": "15", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_function_tracing": "false", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_grace": "80", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_interval": "6", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_peers": "10", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_size": "2000", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_stale": "600", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-06-03T05:24:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_max_size": "100000", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_min_size": "1000", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal_flush_on_shutdown": "true", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal_size": "5120", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_kill_backfill_at": "0", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_loop_before_reset_tphandle": "64", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_cache_size": "50", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_dedup": "true", 2024-06-03T05:24:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_message_max": "40", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_message_max_bytes": "10485760", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_share_max_epochs": "40", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_attr_name_len": "100", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_attr_size": "0", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_backfills": "1", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_markdown_count": "5", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_markdown_period": "600", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_name_len": "2048", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_namespace_len": "256", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_size": "134217728", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-06-03T05:24:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_omap_entries_per_request": "1024", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_blocked_by": "16", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_log_entries": "10000", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pgls": "1024", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_push_cost": "8388608", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_push_objects": "10", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_scrubs": "3", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_trimming_pgs": "2", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_write_op_reply_len": "64", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_write_size": "90", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-06-03T05:24:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_override_recovery_settings": "false", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_profile": "balanced", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-06-03T05:24:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_skip_benchmark": "false", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_base": "805306368", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_cache_min": "134217728", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target": "4294967296", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target_autotune": "false", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_min_pg_log_entries": "250", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_min_recovery_priority": "0", 2024-06-03T05:24:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_heartbeat_interval": "30", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_report_interval": "5", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_report_max_in_flight": "2", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_num_cache_shards": "32", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_num_op_tracker_shard": "32", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_auto_affinity": "true", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_node": "-1", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_prefer_iface": "true", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objecter_finishers": "1", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore": "bluestore", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore_fuse": "false", 2024-06-03T05:24:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore_tracing": "false", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_complaint_time": "30.000000", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_duration": "600", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_size": "20", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_slow_op_size": "20", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_log_threshold": "5", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards": "0", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards_hdd": "5", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards_ssd": "8", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard": "0", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-06-03T05:24:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_pq_min_cost": "65536", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_queue": "mclock_scheduler", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_queue_cut_off": "high", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_thread_suicide_timeout": "150", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_thread_timeout": "15", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_open_classes_on_start": "true", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_os_flags": "0", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_peering_op_priority": "255", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_delete_cost": "1048576", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_delete_priority": "5", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_dups_tracked": "3000", 2024-06-03T05:24:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_trim_max": "10000", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_trim_min": "100", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_object_context_cache_count": "64", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_crimson": "false", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_crush_rule": "-1", 2024-06-03T05:24:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_ec_fast_read": "false", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_bulk": "false", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nodelete": "false", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flags": "0", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_min_size": "0", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pg_num": "32", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pgp_num": "0", 2024-06-03T05:24:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_read_ratio": "70", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_size": "2", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_type": "replicated", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_use_gmt_hitset": "true", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_push_per_object_cost": "1000", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_read_ec_check_for_errors": "false", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recover_clone_overlap": "true", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recover_clone_overlap_limit": "10", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_cost": "20971520", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_delay_start": "0.000000", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active": "0", 2024-06-03T05:24:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active_hdd": "3", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active_ssd": "10", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_chunk": "8388608", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_single_start": "1", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_op_priority": "3", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_op_warn_multiple": "16", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_priority": "5", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_retry_interval": "30.000000", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep": "0.000000", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-06-03T05:24:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_repair_during_recovery": "false", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_requested_scrub_priority": "120", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_rollback_to_cluster_snap": "", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_auto_repair": "false", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_begin_hour": "0", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_begin_week_day": "0", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_chunk_max": "25", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_chunk_min": "5", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_cost": "52428800", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-06-03T05:24:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_during_recovery": "false", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_end_hour": "0", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_end_week_day": "0", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_event_cost": "4096", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_invalid_stats": "true", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_load_threshold": "0.500000", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_max_interval": "604800.000000", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_max_preemptions": "5", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_min_interval": "86400.000000", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_priority": "5", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_sleep": "0.000000", 2024-06-03T05:24:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shutdown_pgref_assert": "false", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_skip_data_digest": "false", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_smart_report_timeout": "5", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_cost": "1048576", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_priority": "5", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep": "0.000000", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_stats_update_period_scrubbing": "15", 2024-06-03T05:24:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_target_transaction_size": "30", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_mode": "writeback", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tracing": "false", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_use_stale_snap": "false", 2024-06-03T05:24:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "osdc_blkin_trace_all": "false", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_kill_at": "0", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_max_join_drift": "10", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_min": "500", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_min_wait": "0.050000", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_propose_interval": "1.000000", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_max": "500", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_max_multiplier": "20", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_min": "10", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_stash_full_interval": "25", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_trim_max": "500", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_trim_min": "250", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "perf": "true", 2024-06-03T05:24:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "-", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrv": "", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "public_bind_addr": "-", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "public_network": "", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "public_network_interface": "", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "qat_compressor_enabled": "false", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "qat_compressor_session_max_number": "256", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_mon_op_timeout": "0", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_osd_op_timeout": "0", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_tracing": "false", 2024-06-03T05:24:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_atime_update_interval": "60", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_balance_parent_reads": "false", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_balance_snap_reads": "false", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blkin_trace_all": "false", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blocklist_expire_seconds": "0", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blocklist_on_break_lock": "true", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache": "true", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_block_writes_upfront": "false", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty": "25165824", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty_object": "0", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_policy": "writearound", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_size": "33554432", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_target_dirty": "16777216", 2024-06-03T05:24:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_clone_copy_on_read": "false", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_compression_hint": "none", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_concurrent_management_ops": "10", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_clone_format": "auto", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_data_pool": "", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_features": "61", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_format": "2", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_map_options": "", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_order": "22", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_pool": "rbd", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_stripe_count": "0", 2024-06-03T05:24:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_stripe_unit": "0", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_disable_zero_copy_writes": "true", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_discard_granularity_bytes": "65536", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_enable_alloc_hint": "true", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_io_scheduler": "simple", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_commit_age": "5.000000", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-06-03T05:24:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_interval": "0", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_order": "24", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_pool": "", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_splay_width": "4", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_localize_parent_reads": "false", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_localize_snap_reads": "false", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-06-03T05:24:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_type": "simple", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_autotune": "true", 2024-06-03T05:24:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_base": "805306368", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_target": "4294967296", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_delete_delay": "0", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_replay_delay": "0", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_to_trash_on_remove": "false", 2024-06-03T05:24:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mtime_update_interval": "60", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_non_blocking_aio": "true", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_op_thread_timeout": "60", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_op_threads": "1", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_parent_cache_enabled": "false", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_mode": "disabled", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_path": "/tmp", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_size": "1073741824", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_plugins": "", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_burst": "0", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_limit": "0", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_exclude_ops": "0", 2024-06-03T05:24:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_burst": "0", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_limit": "0", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_burst": "0", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_limit": "0", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_burst": "0", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_limit": "0", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_schedule_tick_min": "50", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_burst": "0", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-06-03T05:24:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_limit": "0", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_burst": "0", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_limit": "0", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_quiesce_notification_attempts": "10", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_read_from_replica_policy": "default", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_max_bytes": "524288", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_trigger_requests": "10", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_request_timed_out_seconds": "30", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_skip_partial_discard": "true", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_tracing": "false", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_validate_names": "true", 2024-06-03T05:24:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_validate_pool": "true", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "restapi_base_url": "", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "restapi_log_level": "", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_account_default_quota_max_size": "-1", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_acl_grants_max_num": "100", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_admin_entry": "admin", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_backend_store": "rados", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_barbican_url": "", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_beast_enable_async": "true", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_counters_cache": "false", 2024-06-03T05:24:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_index_max_aio": "128", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_quota_ttl": "600", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_enabled": "true", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_expiry_interval": "900", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_lru_size": "25000", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_config_store": "rados", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_content_length_compat": "false", 2024-06-03T05:24:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_copy_obj_progress": "true", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cors_rules_max_num": "100", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cross_domain_policy": "", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_default_encryption_key": "", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_addr": "", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_ca_path": "", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_client_cert": "", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_client_key": "", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_password": "", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_username": "", 2024-06-03T05:24:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_require_ssl": "true", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-06-03T05:24:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_suppress_logs": "true", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_addr": "", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_auth": "token", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_namespace": "", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_prefix": "", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_token_file": "", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_buffersize": "524288", 2024-06-03T05:24:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_low_speed_limit": "1024", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_low_speed_time": "300", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_tcp_keepalive": "0", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_fadvise": "4", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_host": "127.0.0.1", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_port": "6379", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-06-03T05:24:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_changes_size": "1000", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_num_shards": "128", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_window": "30", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_notify_interval_msec": "0", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_sync_poll_interval": "20", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_sync_spawn_window": "20", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_data_log_backing": "fifo", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_region_info_oid": "default.region", 2024-06-03T05:24:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_defer_to_bucket_acls": "", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_res": "500.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-06-03T05:24:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dns_name": "", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dns_s3website_name": "", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dynamic_resharding": "true", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_gc_threads": "true", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_lc_threads": "true", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_ops_log": "false", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_quota_threads": "true", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_static_website": "false", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_usage_log": "false", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enforce_swift_acls": "true", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_exit_timeout_secs": "120", 2024-06-03T05:24:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_expose_bucket": "false", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_extended_http_attrs": "", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_filter": "none", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_frontends": "beast port=7480", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_concurrent_io": "10", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_deferred": "50", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_objs": "32", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_queue_size": "134213632", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_trim_chunk": "16", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_obj_min_wait": "7200", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_processor_max_time": "3600", 2024-06-03T05:24:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_processor_period": "3600", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_get_obj_window_size": "16777216", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_healthcheck_disabling_path": "", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ignore_get_invalid_range": "false", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_init_timeout": "300", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_connection_idle": "300", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_message_timeout": "5000", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_sleep_timeout": "10", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-06-03T05:24:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_domain": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_password": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_password_path": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_project": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_tenant": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_token": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_token_path": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_user": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_api_version": "2", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_domain": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_password": "", 2024-06-03T05:24:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_project": "", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_tenant": "", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_user": "", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_implicit_tenants": "false", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_service_token_enabled": "false", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_token_cache_size": "10000", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_url": "", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_verify_ssl": "true", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_debug_interval": "-1", 2024-06-03T05:24:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_lock_max_time": "90", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_objs": "32", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_rules": "1000", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_worker": "3", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_wp_worker": "3", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_thread_delay": "0", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_dnattr": "uid", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_searchfilter": "", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_uri": "", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-06-03T05:24:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_http_headers": "", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_nonexistent_bucket": "false", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_object_name_utc": "false", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attr_name_len": "0", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attr_size": "0", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attrs_num_in_req": "0", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_chunk_size": "4194304", 2024-06-03T05:24:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_concurrent_requests": "1024", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_dynamic_shards": "1999", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_listing_results": "1000", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_notify_retries": "10", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_objs_per_shard": "100000", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_put_param_size": "1048576", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_put_size": "5368709120", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_slo_entries": "1000", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_md_log_max_shards": "64", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_md_notify_interval_msec": "200", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_meta_sync_poll_interval": "20", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_meta_sync_spawn_window": "20", 2024-06-03T05:24:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_mp_lock_max_time": "600", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multipart_min_part_size": "5242880", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_fhcache_size": "2017", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_lru_lanes": "5", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_max_gc": "300", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_gc_threads": "false", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_lc_threads": "false", 2024-06-03T05:24:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_quota_threads": "false", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_sync_thread": "false", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_num_async_rados_threads": "32", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_num_control_oids": "8", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_numa_node": "-1", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_obj_stripe_size": "4194304", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_chunk_size": "100", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_gc_interval": "600", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_hints_num_shards": "127", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-06-03T05:24:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_thread_timeout": "600", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_tracing": "false", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_token": "", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_url": "", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_verify_ssl": "true", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_rados": "false", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_socket_path": "", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-06-03T05:24:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_push_interval": "2.000000", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_push_interval_max": "30.000000", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_root_pool": ".rgw.root", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_lanes": "3", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_max_buckets": "100", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_partitions": "3", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_print_continue": "true", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_print_prohibited_content_length": "false", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-06-03T05:24:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_tracing": "false", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm": "", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm_id": "", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_region": "", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_region_root_pool": ".rgw.root", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_relaxed_region_enforcement": "false", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-06-03T05:24:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_request_uri": "", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_batch_size": "64", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_max_aio": "128", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_num_logs": "16", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_thread_interval": "600", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_resolve_cname": "false", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rest_getusage_op_compat": "false", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_run_sync_thread": "true", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_keystone": "false", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_ldap": "false", 2024-06-03T05:24:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_rados": "true", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_sts": "false", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_success_create_obj_status": "0", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_scheduler_type": "throttler", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_script_uri": "", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_service_provider_name": "", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_client_id": "", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_client_secret": "", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_entry": "sts", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_key": "sts", 2024-06-03T05:24:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_max_session_duration": "43200", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_min_session_duration": "900", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_token_introspection_url": "", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_account_in_url": "false", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_auth_entry": "auth", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_auth_url": "", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_custom_header": "", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_enforce_content_length": "false", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_need_stats": "true", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_tenant_name": "", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_token_expiration": "86400", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_url": "", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_url_prefix": "swift", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_versioning_enabled": "false", 2024-06-03T05:24:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_lease_period": "120", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_interval": "1200", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_obj_etag_verify": "false", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_history_size": "4096", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_thread_pool_size": "512", 2024-06-03T05:24:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_max_retries": "0", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_require_publish_policy": "false", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_comment": "", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_createby": "", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_encoding": "", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_flag": "false", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_max_size": "5368709120", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_origin": "", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_sha_unit": "524288", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_tracker": "", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_trust_forwarded_https": "false", 2024-06-03T05:24:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_log_tick_interval": "30", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_max_shards": "32", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_max_user_shards": "1", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_use_opa_authz": "false", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_counters_cache": "false", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_counters_cache_size": "10000", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_default_quota_max_size": "-1", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_max_buckets": "1000", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_policies_max_num": "100", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-06-03T05:24:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_interval": "86400", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_unique_email": "true", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_verify_ssl": "true", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_website_routing_rules_max_num": "50", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone": "", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone_id": "", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup": "", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup_id": "", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgwlc_auto_session_clear": "true", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rgwlc_skip_bucket_step": "false", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_block_size": "4096", 2024-06-03T05:24:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_shard_bits": "4", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_size": "536870912", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_type": "binned_lru", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_compaction_stats": "false", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_extended_stats": "false", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_memory_stats": "false", 2024-06-03T05:24:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_index_type": "binary_search", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_log_to_ceph_log": "true", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_metadata_block_size": "4096", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_partition_filters": "false", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_perf": "false", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "rotating_keys_renewal_timeout": "10", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "run_dir": "/var/run/ceph", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_block_create": "true", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_cache_lru_size": "67108864", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_cbjournal_size": "5368709120", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_data_delta_based_overwrite": "0", 2024-06-03T05:24:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_default_max_object_size": "16777216", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_device_size": "53687091200", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_full_integrity_check": "false", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_capacity": "16", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_iodepth_limit": "5", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_main_device_type": "SSD", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_max_concurrent_transactions": "8", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_max_data_allocation_size": "32768", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-06-03T05:24:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_segment_size": "67108864", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "set_keepcaps": "false", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "setgroup": "", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "setuser": "", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "setuser_match_path": "", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_misplaced_ratio": "0.050000", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "thp": "false", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "threadpool_default_timeout": "60", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "threadpool_empty_queue_max_wait": "2", 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout: "throttler_perf_counter": "true" 2024-06-03T05:24:26.455 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:26.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-06-03T05:24:26.762 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a: {} 2024-06-03T05:24:26.762 INFO:teuthology.orchestra.run.smithi038.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-03T05:24:26.874 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b: {} 2024-06-03T05:24:26.874 INFO:teuthology.orchestra.run.smithi038.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-03T05:24:26.977 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c: {} 2024-06-03T05:24:26.977 INFO:teuthology.orchestra.run.smithi038.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-03T05:24:26.988 DEBUG:teuthology.orchestra.run.smithi038:> type ceph-objectstore-tool 2024-06-03T05:24:27.003 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-06-03T05:24:27.003 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-06-03T05:24:27.019 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-06-03T05:24:27.020 INFO:teuthology.run_tasks:Running task mon_recovery... 2024-06-03T05:24:27.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T05:24:27.030 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-06-03T05:24:27.031 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:24:27.031 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:24:27.031 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-06-03T05:24:27.031 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-06-03T05:24:27.031 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:24:27.032 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-06-03T05:24:27.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:24:27.032 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-06-03T05:24:27.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:24:27.051 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:27.049+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:27.141 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:27.139+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:27.206 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:27.206 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:27.207 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:27.207 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:27.215 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:24:27.265 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:27.264+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:27.334 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:27.335+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:27.379 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:24:27.382 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:24:27.435 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:27.436 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:27.436 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:27.436+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:27.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:24:27.516 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:24:27.516 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:24:27.536 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:27.536+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:27.621 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:27.621 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:27.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:24:27.638 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:24:27.639 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:24:27.642 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:27.640+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:27.774 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:24:27.774 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:24:27.774 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:27.775+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:27.790 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:27.790 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:27.800 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:24:27.843 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:27.843+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:27.901 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:24:27.902 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:24:27.945 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:27.943+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:28.001 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:28.002 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:28.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:24:28.044 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:24:28.045 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:24:28.045 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:28.043+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:28.071 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:34.870 35035== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:28.075 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:34.875 35035== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:28.130 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:34.930 35035== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:28.135 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:34.935 35037== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:28.137 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:34.937 35037== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:28.146 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:28.145+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:28.157 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:28.157 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":24,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:24:16.465229+0000","last_up_change":"2024-06-03T05:24:09.927340+0000","last_in_change":"2024-06-03T05:23:16.043190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T05:23:56.753149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-03T05:24:12.898324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":905141127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":905141127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":905141127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":905141127}]},"public_addr":"172.21.15.38:6809/905141127","cluster_addr":"172.21.15.38:6811/905141127","heartbeat_back_addr":"172.21.15.38:6816/905141127","heartbeat_front_addr":"172.21.15.38:6813/905141127","state":["exists","up"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3490615740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3490615740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3490615740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3490615740}]},"public_addr":"172.21.15.38:6810/3490615740","cluster_addr":"172.21.15.38:6812/3490615740","heartbeat_back_addr":"172.21.15.38:6815/3490615740","heartbeat_front_addr":"172.21.15.38:6814/3490615740","state":["exists","up"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":3359326199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3359326199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3359326199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":3359326199}]},"public_addr":"172.21.15.38:6805/3359326199","cluster_addr":"172.21.15.38:6806/3359326199","heartbeat_back_addr":"172.21.15.38:6808/3359326199","heartbeat_front_addr":"172.21.15.38:6807/3359326199","state":["exists","up"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":2289724788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2289724788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2289724788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2289724788}]},"public_addr":"172.21.15.38:6801/2289724788","cluster_addr":"172.21.15.38:6802/2289724788","heartbeat_back_addr":"172.21.15.38:6804/2289724788","heartbeat_front_addr":"172.21.15.38:6803/2289724788","state":["exists","up"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3833397294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":3833397294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":3833397294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3833397294}]},"public_addr":"172.21.15.64:6808/3833397294","cluster_addr":"172.21.15.64:6809/3833397294","heartbeat_back_addr":"172.21.15.64:6811/3833397294","heartbeat_front_addr":"172.21.15.64:6810/3833397294","state":["exists","up"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3915827095}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":3915827095}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":3915827095}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3915827095}]},"public_addr":"172.21.15.64:6800/3915827095","cluster_addr":"172.21.15.64:6801/3915827095","heartbeat_back_addr":"172.21.15.64:6803/3915827095","heartbeat_front_addr":"172.21.15.64:6802/3915827095","state":["exists","up"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1144336651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1144336651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6807","nonce":1144336651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1144336651}]},"public_addr":"172.21.15.64:6804/1144336651","cluster_addr":"172.21.15.64:6805/1144336651","heartbeat_back_addr":"172.21.15.64:6807/1144336651","heartbeat_front_addr":"172.21.15.64:6806/1144336651","state":["exists","up"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":370645108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":370645108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":370645108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":370645108}]},"public_addr":"172.21.15.64:6812/370645108","cluster_addr":"172.21.15.64:6813/370645108","heartbeat_back_addr":"172.21.15.64:6815/370645108","heartbeat_front_addr":"172.21.15.64:6814/370645108","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:56.081253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.642638+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.131979+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.147738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.907705+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.533669+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.310102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.959957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:24:28.158 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:34.903 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:28.160 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:34.905 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:28.160 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:28.160 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:28.175 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:24:28.177 INFO:tasks.mon_recovery.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-03T05:23:56.753149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-06-03T05:24:12.898324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-03T05:24:28.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-03T05:24:28.194 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:34.993 35037== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:28.195 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:24:28.195 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:24:28.206 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:34.951 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:28.241 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:28.242+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:28.340 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:24:28.340 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:24:28.347 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:28.346+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:28.365 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:28.365 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:28.374 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:24:28.443 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:28.444+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:28.544 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:28.544+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:28.549 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:28.549 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:28.560 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:24:28.646 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:28.646+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:28.696 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:28.697 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:28.706 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:24:28.750 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:28.750+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:28.833 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-06-03T05:24:28.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-06-03T05:24:28.848 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:28.849+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:28.854 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:28.855 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:28.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:24:28.950 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:28.950+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:29.046 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:29.046 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:29.050 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:29.050+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:29.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:24:29.150 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:29.151+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:29.230 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:29.230 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:29.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:24:29.284 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:29.284+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:29.353 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:29.354+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:29.382 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:29.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:24:29.432 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:24:29.474 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 8 2024-06-03T05:24:29.475 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:29.475+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:29.487 INFO:tasks.mon_recovery:mon ids = ['a', 'c', 'b'] 2024-06-03T05:24:29.487 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-06-03T05:24:29.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:24:29.577 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:29.577+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:29.586 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:29.586 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:29.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:24:29.656 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:29.656+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:29.733 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:29.733 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:29.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:24:29.756 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:29.756+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:29.857 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:29.858+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:29.924 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:29.924 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:29.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:24:29.975 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:29.975+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:30.060 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:30.060+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:30.108 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:30.108 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:30.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:24:30.160 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:30.160+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:30.216 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":80,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:24:30.229 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-06-03T05:24:30.229 INFO:tasks.mon_recovery:verifying all monitors are in the quorum 2024-06-03T05:24:30.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-06-03T05:24:30.253 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:30.253 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:30.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:24:30.272 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:30.271+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:30.353 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:24:30.399 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:30.399+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:30.411 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:30.411 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:30.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "a", 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 0, 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "leader", 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: "election_epoch": 8, 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 88673, 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum": [ 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: 2 2024-06-03T05:24:30.456 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum_age": 80, 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "features": { 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "required_con": "2450521708007866372", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "required_mon": [ 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "kraken", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "luminous", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "mimic", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap-prune", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "nautilus", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "elector-pinging", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy", 2024-06-03T05:24:30.457 INFO:teuthology.orchestra.run.smithi038.stdout: "reef", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "squid" 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum_con": "4540701547738038271", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum_mon": [ 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "kraken", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "luminous", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "mimic", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap-prune", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "nautilus", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "elector-pinging", 2024-06-03T05:24:30.458 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy", 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "reef", 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "squid" 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "outside_quorum": [], 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "extra_probe_peers": [], 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_provider": [], 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "monmap": { 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 1, 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "ebc09f7e-f094-4421-826f-479fc526fc62", 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-06-03T05:22:16.844560Z", 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-06-03T05:22:16.844560Z", 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "min_mon_release": 19, 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "min_mon_release_name": "squid", 2024-06-03T05:24:30.459 INFO:teuthology.orchestra.run.smithi038.stdout: "election_strategy": 1, 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "disallowed_leaders: ": "", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": false, 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "tiebreaker_mon": "", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_ranks: ": "", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "features": { 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "persistent": [ 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "kraken", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "luminous", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "mimic", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap-prune", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "nautilus", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific", 2024-06-03T05:24:30.460 INFO:teuthology.orchestra.run.smithi038.stdout: "elector-pinging", 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy", 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "reef", 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "squid" 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "optional": [] 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "mons": [ 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 0, 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "a", 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:3300", 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6789", 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6789/0", 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6789/0", 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: "priority": 0, 2024-06-03T05:24:30.462 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "{}" 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 1, 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "b", 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:3300", 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6789", 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6789/0", 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6789/0", 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "priority": 0, 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "{}" 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 2, 2024-06-03T05:24:30.464 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "c", 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:3301", 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6790", 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6790/0", 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6790/0", 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: "priority": 0, 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "{}" 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: "feature_map": { 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: "mon": [ 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.466 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 1 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": [ 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 2 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: "client": [ 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.467 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.468 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 4 2024-06-03T05:24:30.468 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.468 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.468 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": false 2024-06-03T05:24:30.468 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:30.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-06-03T05:24:30.470 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:30.470+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:30.559 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:30.559 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:30.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:24:30.609 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:30.609+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:30.616 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:24:30.616 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:24:30.672 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:30.672+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "c", 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 2, 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "peon", 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: "election_epoch": 8, 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 88907, 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum": [ 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: 2 2024-06-03T05:24:30.685 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum_age": 80, 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "features": { 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "required_con": "2450521708007866372", 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "required_mon": [ 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "kraken", 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "luminous", 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "mimic", 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap-prune", 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "nautilus", 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus", 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific", 2024-06-03T05:24:30.686 INFO:teuthology.orchestra.run.smithi038.stdout: "elector-pinging", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "reef", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "squid" 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum_con": "4540701547738038271", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum_mon": [ 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "kraken", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "luminous", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "mimic", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap-prune", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "nautilus", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus", 2024-06-03T05:24:30.687 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific", 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "elector-pinging", 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy", 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "reef", 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "squid" 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "outside_quorum": [], 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "extra_probe_peers": [], 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_provider": [], 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "monmap": { 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 1, 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "ebc09f7e-f094-4421-826f-479fc526fc62", 2024-06-03T05:24:30.688 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-06-03T05:22:16.844560Z", 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-06-03T05:22:16.844560Z", 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "min_mon_release": 19, 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "min_mon_release_name": "squid", 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "election_strategy": 1, 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "disallowed_leaders: ": "", 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": false, 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "tiebreaker_mon": "", 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_ranks: ": "", 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "features": { 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "persistent": [ 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "kraken", 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "luminous", 2024-06-03T05:24:30.689 INFO:teuthology.orchestra.run.smithi038.stdout: "mimic", 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap-prune", 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "nautilus", 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus", 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific", 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "elector-pinging", 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy", 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "reef", 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "squid" 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "optional": [] 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.690 INFO:teuthology.orchestra.run.smithi038.stdout: "mons": [ 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 0, 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "a", 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:3300", 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-06-03T05:24:30.691 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6789", 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6789/0", 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6789/0", 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: "priority": 0, 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "{}" 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 1, 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "b", 2024-06-03T05:24:30.692 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:3300", 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6789", 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6789/0", 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6789/0", 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "priority": 0, 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "{}" 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 2, 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "c", 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.694 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:3301", 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6790", 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6790/0", 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6790/0", 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: "priority": 0, 2024-06-03T05:24:30.695 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "{}" 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: "feature_map": { 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: "mon": [ 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 1 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": [ 2024-06-03T05:24:30.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 1 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client": [ 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 1 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.697 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": [ 2024-06-03T05:24:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 1 2024-06-03T05:24:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.698 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": false 2024-06-03T05:24:30.698 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:30.699 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-06-03T05:24:30.713 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:24:30.713 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:24:30.748 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:30.748 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:30.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:24:30.799 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:30.799+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:30.819 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:24:30.819 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:24:30.874 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:30.874+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:30.899 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "b", 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 1, 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "peon", 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "election_epoch": 8, 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 89118, 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum": [ 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: 2 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum_age": 80, 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "features": { 2024-06-03T05:24:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "required_con": "2450521708007866372", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "required_mon": [ 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "kraken", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "luminous", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mimic", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap-prune", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "nautilus", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "elector-pinging", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "reef", 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "squid" 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum_con": "4540701547738038271", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "quorum_mon": [ 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "kraken", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "luminous", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mimic", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap-prune", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "nautilus", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "elector-pinging", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "reef", 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "squid" 2024-06-03T05:24:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "outside_quorum": [], 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "extra_probe_peers": [], 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "sync_provider": [], 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "monmap": { 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 1, 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "ebc09f7e-f094-4421-826f-479fc526fc62", 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-06-03T05:22:16.844560Z", 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-06-03T05:22:16.844560Z", 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "min_mon_release": 19, 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "min_mon_release_name": "squid", 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "election_strategy": 1, 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "disallowed_leaders: ": "", 2024-06-03T05:24:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": false, 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "tiebreaker_mon": "", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_ranks: ": "", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "features": { 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "persistent": [ 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "kraken", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "luminous", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mimic", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap-prune", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "nautilus", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "elector-pinging", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy", 2024-06-03T05:24:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "reef", 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "squid" 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "optional": [] 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mons": [ 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 0, 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "a", 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:3300", 2024-06-03T05:24:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6789", 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6789/0", 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6789/0", 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "priority": 0, 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:24:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "{}" 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 1, 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "b", 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:3300", 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-06-03T05:24:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6789", 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6789/0", 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6789/0", 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "priority": 0, 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "{}" 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "rank": 2, 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "c", 2024-06-03T05:24:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:3301", 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6790", 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 0 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6790/0", 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6790/0", 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "priority": 0, 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "{}" 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "feature_map": { 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mon": [ 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 1 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": [ 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 5 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "client": [ 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 4 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": [ 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "features": "0x3f03cffffffdffff", 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "release": "luminous", 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 1 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": false 2024-06-03T05:24:30.912 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:30.913 INFO:tasks.mon_recovery:restarting each monitor in turn 2024-06-03T05:24:30.914 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-03T05:24:30.914 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:24:30.921 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:24:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:24:30.938 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-06-03T05:24:30.925+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35002) UID: 0 2024-06-03T05:24:30.938 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-06-03T05:24:30.927+0000 9f9e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-03T05:24:30.945 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:30.945 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:30.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:24:30.996 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:30.996+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:31.024 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:24:31.024 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:24:31.131 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:24:31.131 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:24:31.131 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:31.131+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:31.135 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:31.136 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:31.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:24:31.190 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:31.190+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:31.230 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:24:31.230 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:24:31.279 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:31.279+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:31.334 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:24:31.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:24:31.354 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:24:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:24:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:24:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:31.354 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.469500463999999, 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:31.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:31.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:24:31.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:24:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:24:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:24:31.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:31.359 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:31.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:24:31.405 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:31.405+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:31.448 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:24:31.482 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:31.483+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:31.522 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:38.321 35035== 2024-06-03T05:24:31.522 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:38.321 35035== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:24:31.534 INFO:tasks.ceph.mon.a.smithi038.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:24:31.536 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:31.537 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:31.537 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:31.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:24:31.586 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:31.587+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:31.687 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:31.688+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:31.723 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:31.723 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:31.723 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:31.723 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:31.723 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:31.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:24:31.788 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:31.788+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:31.869 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:31.869 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:31.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:31.869 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:24:31.869 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:31.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:24:31.889 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:31.889+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:31.990 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:31.990+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:32.031 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:24:32.031 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:24:32.057 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:32.058 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:32.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:24:32.109 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:32.109+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:32.192 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:32.192+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:32.242 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:32.243 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:32.243 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:32.243 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:32.243 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:32.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:24:32.296 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:32.296+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:32.394 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:32.394+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.572173378999999, 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:24:32.400 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:24:32.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.404 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.313341575999999, 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:24:32.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:24:32.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:32.409 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:32.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:32.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:32.409 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:32.409 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:32.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:24:32.495 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:32.495+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:32.588 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:32.588 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:32.588 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:32.588 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:32.595 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:32.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:32.597 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:24:32.697 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:32.697+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:32.775 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:32.775 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:32.775 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:32.775 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:32.775 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:32.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:24:32.798 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:32.798+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:32.899 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:32.899+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:32.961 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:32.961 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:32.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:32.961 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:24:32.961 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:32.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:24:33.016 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:33.016+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:33.101 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:33.101+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:33.109 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:33.109 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:33.109 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:33.109 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:33.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:24:33.201 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:33.202+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:33.295 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:33.296 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:33.302 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:33.303+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:33.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:24:33.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:24:33.404 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:33.404+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:33.456 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.780254254000001, 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:24:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:24:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:33.461 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:33.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:24:33.509 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:33.509+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:33.603 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:24:33.604 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:24:33.605 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:33.605+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:33.615 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:33.615 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:33.615 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:24:33.615 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:33.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:24:33.698 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:24:33.698 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:24:33.705 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:33.706+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:33.791 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:24:33.791 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:24:33.800 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:33.800 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:33.800 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:24:33.800 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:24:33.800 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:33.806 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:33.807+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:33.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:24:33.884 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:24:33.885 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:24:33.907 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:33.907+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:33.978 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:24:33.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:24:34.008 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:34.008+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:34.018 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:34.018 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:34.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:24:34.073 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:24:34.073 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:24:34.112 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:34.112+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:34.174 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:24:34.174 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:24:34.210 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:34.210+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:34.219 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:34.219 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:34.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:24:34.280 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:24:34.281 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:24:34.335 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:34.335+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:34.402 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:34.402 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:34.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:24:34.412 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:34.412+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:34.516 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:34.515+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:34.547 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:34.547 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:34.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-06-03T05:24:34.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:24:34.641 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:34.641+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:34.714 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:34.715+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:34.729 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:34.730 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:34.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:24:34.815 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:34.815+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:34.916 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:34.916+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:34.922 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:34.922 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:34.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:24:35.017 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:35.017+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:35.064 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:35.064 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:35.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:24:35.117 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:35.117+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:35.218 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:35.218+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:35.247 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:35.247 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:35.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:24:35.318 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:35.318+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:35.389 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:35.390 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:35.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:24:35.442 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:35.442+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:35.520 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:35.520+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:35.533 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:35.533 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:35.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:24:35.621 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:35.621+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:35.693 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:35.693 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:35.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:24:35.722 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:35.722+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:35.822 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:35.823+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:35.889 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:35.889 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:35.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:24:35.923 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:35.923+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:36.024 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:36.024+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:36.033 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:36.033 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:36.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:24:36.124 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:36.125+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:36.176 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:36.176 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:36.185 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:24:36.228 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:36.228+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:36.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:24:36.322 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:36.322 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:36.327 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:36.327+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:36.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:24:36.427 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:36.428+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:36.509 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:36.509 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:36.519 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:24:36.528 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:36.528+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:36.629 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:36.629+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:36.694 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:36.694 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:36.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:24:36.730 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:36.730+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:36.831 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:36.831+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:36.840 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:36.840 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:36.849 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:24:36.931 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:36.931+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:36.986 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:36.986 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:36.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:24:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:24:37.016 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-06-03T05:24:37.017 INFO:tasks.ceph.mon.a:Stopped 2024-06-03T05:24:37.017 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-06-03T05:24:37.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:24:37.032 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:24:37.032 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:24:37.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:24:37.033 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:37.033+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:37.134 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:37.134+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:37.177 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:37.177 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:37.186 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:24:37.234 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:37.234+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:37.335 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:37.335+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:37.360 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:37.360 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:37.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:24:37.436 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:37.436+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:37.504 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:37.505 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:37.514 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:24:37.537 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:37.537+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:24:37.639 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:24:37.639 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:37.638+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:37.649 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:37.649 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:37.651 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2024-06-03T05:24:36.305322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218608,"kb_used_data":2448,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756112,"statfs":{"total":773094113280,"available":772870258688,"internally_reserved":0,"allocated":2506752,"data_stored":1400056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001721"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T05:24:29.415913+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:24:29.415913+0000","last_peered":"2024-06-03T05:24:29.415913+0000","last_clean":"2024-06-03T05:24:29.415913+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:24:29.415913+0000","last_undegraded":"2024-06-03T05:24:29.415913+0000","last_fullsized":"2024-06-03T05:24:29.415913+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T05:24:29.581410+0000","last_change":"2024-06-03T05:24:15.538413+0000","last_active":"2024-06-03T05:24:29.581410+0000","last_peered":"2024-06-03T05:24:29.581410+0000","last_clean":"2024-06-03T05:24:29.581410+0000","last_became_active":"2024-06-03T05:24:15.511120+0000","last_became_peered":"2024-06-03T05:24:15.511120+0000","last_unstale":"2024-06-03T05:24:29.581410+0000","last_undegraded":"2024-06-03T05:24:29.581410+0000","last_fullsized":"2024-06-03T05:24:29.581410+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:05:38.432196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T05:24:29.411343+0000","last_change":"2024-06-03T05:24:14.763971+0000","last_active":"2024-06-03T05:24:29.411343+0000","last_peered":"2024-06-03T05:24:29.411343+0000","last_clean":"2024-06-03T05:24:29.411343+0000","last_became_active":"2024-06-03T05:24:14.758379+0000","last_became_peered":"2024-06-03T05:24:14.758379+0000","last_unstale":"2024-06-03T05:24:29.411343+0000","last_undegraded":"2024-06-03T05:24:29.411343+0000","last_fullsized":"2024-06-03T05:24:29.411343+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:01:23.148286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T05:24:29.577328+0000","last_change":"2024-06-03T05:24:15.223666+0000","last_active":"2024-06-03T05:24:29.577328+0000","last_peered":"2024-06-03T05:24:29.577328+0000","last_clean":"2024-06-03T05:24:29.577328+0000","last_became_active":"2024-06-03T05:24:15.216506+0000","last_became_peered":"2024-06-03T05:24:15.216506+0000","last_unstale":"2024-06-03T05:24:29.577328+0000","last_undegraded":"2024-06-03T05:24:29.577328+0000","last_fullsized":"2024-06-03T05:24:29.577328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:00:18.708313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T05:24:29.405442+0000","last_change":"2024-06-03T05:24:15.207972+0000","last_active":"2024-06-03T05:24:29.405442+0000","last_peered":"2024-06-03T05:24:29.405442+0000","last_clean":"2024-06-03T05:24:29.405442+0000","last_became_active":"2024-06-03T05:24:15.202389+0000","last_became_peered":"2024-06-03T05:24:15.202389+0000","last_unstale":"2024-06-03T05:24:29.405442+0000","last_undegraded":"2024-06-03T05:24:29.405442+0000","last_fullsized":"2024-06-03T05:24:29.405442+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:55:27.456089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T05:24:29.458044+0000","last_change":"2024-06-03T05:24:14.681733+0000","last_active":"2024-06-03T05:24:29.458044+0000","last_peered":"2024-06-03T05:24:29.458044+0000","last_clean":"2024-06-03T05:24:29.458044+0000","last_became_active":"2024-06-03T05:24:14.652965+0000","last_became_peered":"2024-06-03T05:24:14.652965+0000","last_unstale":"2024-06-03T05:24:29.458044+0000","last_undegraded":"2024-06-03T05:24:29.458044+0000","last_fullsized":"2024-06-03T05:24:29.458044+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:38:49.655223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T05:24:29.397973+0000","last_change":"2024-06-03T05:24:14.688098+0000","last_active":"2024-06-03T05:24:29.397973+0000","last_peered":"2024-06-03T05:24:29.397973+0000","last_clean":"2024-06-03T05:24:29.397973+0000","last_became_active":"2024-06-03T05:24:14.659250+0000","last_became_peered":"2024-06-03T05:24:14.659250+0000","last_unstale":"2024-06-03T05:24:29.397973+0000","last_undegraded":"2024-06-03T05:24:29.397973+0000","last_fullsized":"2024-06-03T05:24:29.397973+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:36:21.716172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T05:24:29.403027+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:24:29.403027+0000","last_peered":"2024-06-03T05:24:29.403027+0000","last_clean":"2024-06-03T05:24:29.403027+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:24:29.403027+0000","last_undegraded":"2024-06-03T05:24:29.403027+0000","last_fullsized":"2024-06-03T05:24:29.403027+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T05:24:29.600399+0000","last_change":"2024-06-03T05:24:12.154052+0000","last_active":"2024-06-03T05:24:29.600399+0000","last_peered":"2024-06-03T05:24:29.600399+0000","last_clean":"2024-06-03T05:24:29.600399+0000","last_became_active":"2024-06-03T05:24:11.479785+0000","last_became_peered":"2024-06-03T05:24:11.479785+0000","last_unstale":"2024-06-03T05:24:29.600399+0000","last_undegraded":"2024-06-03T05:24:29.600399+0000","last_fullsized":"2024-06-03T05:24:29.600399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:11:30.587750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378638,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":75547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411339,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":75547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":75547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":75547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411342,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":473387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476747,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":75547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":75547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":473387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:24:37.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:24:37.719 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":87,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:24:37.739 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:37.739+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:37.835 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:37.835 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:37.840 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:37.840+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:37.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:24:37.941 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:37.941+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:38.018 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:38.019 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:38.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:24:38.072 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:38.072+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:38.144 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:38.144+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:38.161 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:38.162 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:38.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:24:38.255 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:38.255+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:38.345 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:38.345 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:38.346 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:38.346+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:38.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:24:38.447 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:38.447+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:38.527 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:38.527 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:38.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:24:38.580 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:38.580+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:38.649 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:38.649+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:38.669 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:38.669 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:38.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:24:38.763 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:38.763+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:38.851 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:38.851+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:38.852 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:38.852 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:38.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:24:38.952 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:38.952+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:39.040 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:39.040 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:39.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:24:39.053 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:39.053+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:39.154 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:39.154+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:39.226 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:39.226 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:39.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:24:39.255 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:39.255+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:39.356 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:39.356+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:39.409 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:39.409 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:39.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:24:39.460 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:39.459+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:39.558 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:39.559+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:39.592 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:39.592 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:39.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:24:39.659 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:39.659+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:39.751 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:39.751 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:39.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:24:39.761 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:39.761+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:39.861 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:39.862+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:39.936 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:39.936 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:39.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:24:39.961 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:39.962+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:40.063 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:40.063+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:40.081 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:40.081 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:40.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:24:40.164 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:40.165+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:40.224 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:40.224 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:40.233 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:24:40.267 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:40.267+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:40.368 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:40.368+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:40.369 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:40.369 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:40.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:24:40.471 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:40.472+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:40.514 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:40.514 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:40.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:24:40.572 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:40.572+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:40.660 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:40.660 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:40.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:24:40.675 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:40.675+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:40.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:24:40.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-06-03T05:24:40.796 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:40.796+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:40.861 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:40.861 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:40.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:24:40.914 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:40.913+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:40.977 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:40.977+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:41.046 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:41.046 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:41.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:24:41.078 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:41.078+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:41.179 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:41.180+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:41.194 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:41.194 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:41.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:24:41.280 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:41.281+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:41.379 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:41.380 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:41.381 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:41.381+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:41.389 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:24:41.482 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:41.482+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:41.565 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:41.565 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:41.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:24:41.619 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:41.619+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:41.683 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:41.683+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:41.721 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:41.722 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:41.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:24:41.816 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:41.816+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:41.885 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:41.885+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:41.917 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:41.917 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:41.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:24:41.986 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:41.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:42.087 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:42.087+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:42.100 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:42.100 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:42.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:24:42.196 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:42.196+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:42.286 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:42.286 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:42.290 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:42.290+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:42.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:24:42.390 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:42.390+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:42.467 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:42.467 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:42.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:24:42.490 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:42.491+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:42.592 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:42.592+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:42.648 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:42.649 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:42.653 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:24:42.653 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:24:42.653 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-06-03T05:24:42.653 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-06-03T05:24:42.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:24:42.700 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:42.699+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:42.794 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:42.794+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:42.833 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:42.833 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:42.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:24:42.894 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:42.894+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:42.977 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:42.977 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:42.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:24:43.030 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:43.029+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:43.096 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:43.096+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:43.165 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:43.165 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:43.174 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:24:43.197 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:43.198+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:43.298 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:43.298+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:43.307 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:43.308 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:43.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:24:43.402 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:43.402+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:43.492 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:43.492 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:43.499 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:43.499+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:43.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:24:43.601 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:43.601+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:43.682 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:43.682 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:43.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:24:43.733 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:43.733+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:43.803 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:43.803+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:43.838 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:43.838 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:43.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:24:43.933 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:43.933+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:44.005 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:44.005+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:44.022 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:44.022 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:44.031 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:24:44.106 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:44.106+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:44.204 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:44.204 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:44.206 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:44.206+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:44.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:24:44.307 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:44.307+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:44.388 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:44.389 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:44.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:24:44.440 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:44.440+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:44.509 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:44.509+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:44.572 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:44.572 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:44.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:24:44.609 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:44.609+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:44.710 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:44.711+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:44.715 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:44.715 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:44.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:24:44.812 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:44.812+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:44.900 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:44.900 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:44.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:24:44.912 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:44.912+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:45.033 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:45.033+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:45.047 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:45.047 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:45.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:24:45.114 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:45.114+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:45.215 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:45.215+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:45.230 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:45.230 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:45.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:24:45.325 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:45.325+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:45.416 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:45.417 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:45.417 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:45.417+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:45.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:24:45.518 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:45.518+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:45.602 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:45.602 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:45.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:24:45.655 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:45.654+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:45.719 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:45.719+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:45.789 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:45.789 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:45.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:24:45.845 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:45.845+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:45.920 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:45.919+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:45.940 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:45.940 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:45.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:24:46.020 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:46.021+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:46.122 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:46.121+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:46.124 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:46.124 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:46.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:24:46.222 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:46.222+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:46.324 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:46.323+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:46.332 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:46.332 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:46.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:24:46.402 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:24:46.402 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:24:46.426 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:46.425+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:46.503 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:24:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:24:46.519 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:46.520 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:46.525 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:46.524+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:46.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:24:46.540 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-06-03T05:24:46.540 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-06-03T05:24:46.540 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-06-03T05:24:46.540 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-06-03T05:24:46.540 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-06-03T05:24:46.540 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-06-03T05:24:46.541 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-06-03T05:24:46.541 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:24:46.541 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-06-03T05:24:46.541 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-06-03T05:24:46.541 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-06-03T05:24:46.541 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-06-03T05:24:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:24:46.544 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-03T05:24:46.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-06-03T05:24:46.603 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:24:46.604 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:24:46.626 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:46.627+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:46.706 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:46.707 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:46.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:24:46.718 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:24:46.719 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:24:46.727 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:46.728+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:46.828 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:46.829+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:46.853 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:24:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:24:46.854 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:46.854 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:46.864 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:24:46.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:24:46.921 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:24:46.929 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:46.928+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:46.935 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-06-03T05:24:46.935 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-03T05:24:46.935 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-06-03T05:24:46.948 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:24:46.949 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:24:46.950 INFO:tasks.ceph.mon.a:Started 2024-06-03T05:24:46.950 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-06-03T05:24:46.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:24:47.030 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:47.030+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:47.061 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:24:47.062 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:24:47.066 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:24:47.066 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:47.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:24:47.097 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-06-03T05:24:47.098 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-06-03T05:24:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:24:47.102 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-03T05:24:47.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-06-03T05:24:47.145 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:47.144+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:47.161 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:24:47.161 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:24:47.232 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:47.232+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:47.251 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:24:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:24:47.252 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:47.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:24:47.333 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:47.333+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:47.370 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-06-03T05:24:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-06-03T05:24:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:24:47.374 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-03T05:24:47.374 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-06-03T05:24:47.397 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:24:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:24:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:24:47.397 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:47.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:24:47.459 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:47.459+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:47.534 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:47.534+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:47.544 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:47.544 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:47.544 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:24:47.545 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:47.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:24:47.635 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:47.635+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:47.655 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-06-03T05:24:47.655 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-06-03T05:24:47.656 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-06-03T05:24:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:24:47.659 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-03T05:24:47.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-06-03T05:24:47.669 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:24:47.729 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:47.729 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:47.729 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:24:47.730 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:24:47.730 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:47.736 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:47.737+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:47.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:24:47.837 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:47.838+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:47.892 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:47.892 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:24:47.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:24:47.892 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:24:47.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.022385216, 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:24:47.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:47.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:47.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:24:47.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:24:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:47.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:47.897 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:47.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:24:47.937 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-06-03T05:24:47.937 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-06-03T05:24:47.937 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-06-03T05:24:47.937 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-06-03T05:24:47.937 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-06-03T05:24:47.937 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-06-03T05:24:47.937 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-06-03T05:24:47.937 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:24:47.938 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-06-03T05:24:47.938 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-06-03T05:24:47.938 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-06-03T05:24:47.938 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-06-03T05:24:47.938 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:47.936+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:24:47.940 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-03T05:24:47.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-06-03T05:24:48.038 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:48.038+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:48.062 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:48.062 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:48.062 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:48.062 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:48.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:24:48.138 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:48.137+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:48.201 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-06-03T05:24:48.202 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-06-03T05:24:48.204 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:24:48.205 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-03T05:24:48.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-06-03T05:24:48.250 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:48.250+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:48.276 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:48.276 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:48.276 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:48.276 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:48.276 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:48.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:24:48.341 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:48.340+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:48.442 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:48.441+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:48.445 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-06-03T05:24:48.446 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-06-03T05:24:48.448 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:24:48.449 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-03T05:24:48.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-06-03T05:24:48.483 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:48.483 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:48.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:48.483 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:24:48.483 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:48.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:24:48.580 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:48.580+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:48.644 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:48.644+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:48.671 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-06-03T05:24:48.671 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-06-03T05:24:48.672 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-06-03T05:24:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:24:48.676 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-03T05:24:48.691 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:48.691 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:48.691 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:48.691 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:48.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:24:48.744 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:48.744+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:48.866 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:48.866+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:48.880 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:48.880 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:48.881 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:48.881 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:48.881 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:48.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:24:48.890 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:24:48.975 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:48.975+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:49.047 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:49.047+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:49.088 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.271008965999997, 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:24:49.089 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:49.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:24:49.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:24:49.093 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.012177162999997, 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:24:49.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:24:49.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:49.097 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:49.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:49.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:49.098 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:49.098 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:49.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:24:49.147 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:49.146+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:49.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:24:49.249 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:49.248+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:49.275 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:49.276 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:49.276 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:49.276 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:49.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:24:49.353 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:49.352+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:49.449 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:24:49.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:24:49.451 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:49.450+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:49.459 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:49.460 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:49.460 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:49.460 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:49.460 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:49.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:24:49.548 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:24:49.549 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:24:49.550 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:49.549+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:49.651 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:49.652+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:49.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:24:49.727 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:24:49.728 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:49.728 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:49.738 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:24:49.829 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:24:49.830 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:24:49.830 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:49.829+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:49.853 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:49.852+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:49.916 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:49.916 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:49.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:24:49.927 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:24:49.927 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:24:49.970 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:49.970+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:50.025 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:24:50.025 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:24:50.055 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:50.054+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:50.086 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:50.086 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:50.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:24:50.135 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:24:50.135 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:24:50.156 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:50.155+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:50.178 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:24:50.250 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:24:50.250 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:24:50.257 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:50.257+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:50.287 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:50.287 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:50.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:24:50.377 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:50.373+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:50.434 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:50.434 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:50.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:24:50.488 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:50.487+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:50.560 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:50.560+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:50.577 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:50.577 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:50.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:24:50.660 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:50.661+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:50.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:24:50.761 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:50.761+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:50.762 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:50.762 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:50.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:24:50.862 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:50.861+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:50.907 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:50.907 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:50.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:24:50.963 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:50.963+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:51.064 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:51.064+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:51.089 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:51.089 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:51.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:24:51.164 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:51.165+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:51.236 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:51.236 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:51.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:24:51.289 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:51.288+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:51.367 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:51.366+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:51.379 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:51.379 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:51.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:24:51.437 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:24:51.494 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:51.494+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:51.563 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:51.563 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:51.568 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:51.569+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:51.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:24:51.693 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:51.692+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:51.708 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:51.708 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:51.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:24:51.802 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:51.801+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:51.889 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:51.888+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:51.904 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:51.905 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:51.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:24:51.973 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:51.973+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:52.074 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:52.073+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:52.089 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:52.089 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:52.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:24:52.183 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:52.182+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:52.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:24:52.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:24:52.275 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:52.275 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:52.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:24:52.309 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:52.308+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:52.377 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:52.377+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:52.449 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:52.449 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:52.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:24:52.501 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:52.500+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:52.559 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:24:52.559 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:24:52.578 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:52.577+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:52.639 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:52.639 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:52.639 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:52.639 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:52.639 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:52.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:24:52.679 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:52.679+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:52.704 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:24:52.704 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:24:52.780 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:52.780+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:52.789 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:52.789 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:52.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:52.789 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:24:52.789 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:52.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:24:52.820 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:24:52.820 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:24:52.881 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:52.880+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:52.947 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:52.947 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:52.947 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:52.947 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:52.954 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:24:52.954 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:24:52.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:24:52.982 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:52.981+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:53.049 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:24:53.049 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:24:53.083 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:53.084+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:53.135 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:53.136 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:53.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:24:53.147 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:24:53.148 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:24:53.184 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:53.184+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:53.240 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:24:53.240 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:24:53.285 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:53.284+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:53.307 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:53.307 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:53.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:53.307 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:24:53.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.307 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:24:53.307 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.489746910999997, 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:24:53.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:24:53.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:24:53.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.230915107999998, 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:24:53.311 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:53.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:24:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:24:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:24:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:53.315 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:53.315 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:53.319 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:24:53.337 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:24:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:24:53.395 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:53.395+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:53.445 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:24:53.458 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:53.458 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:53.458 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:53.458 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:53.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:24:53.486 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:53.485+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:53.587 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:53.587+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:53.658 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:53.659 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:53.659 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:53.659 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:53.659 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:53.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:24:53.676 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:24:53.676 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:24:53.711 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:53.710+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:53.788 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:53.788+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:53.848 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:53.848 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:53.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:53.848 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:24:53.848 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:53.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:24:53.889 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:53.889+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:53.990 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:53.990+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:54.033 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:54.033 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:54.033 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:24:54.033 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:54.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:24:54.092 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:54.092+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:54.178 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:54.178 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:24:54.178 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:24:54.178 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:24:54.179 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:54.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:24:54.193 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:54.192+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:54.294 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:54.294+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:54.365 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:24:54.365 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:24:54.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:24:54.365 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:24:54.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:54.365 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:24:54.365 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:24:54.365 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.700102254000001, 2024-06-03T05:24:54.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:24:54.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:24:54.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:54.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:24:54.369 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:24:54.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:24:54.418 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:54.417+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:54.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:24:54.499 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:54.499+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:54.518 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:54.518 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:54.518 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:24:54.518 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:54.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:24:54.613 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:54.613+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:54.727 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:54.702+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:54.745 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:54.745 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:54.746 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:24:54.746 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:24:54.746 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:54.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:24:54.798 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:54.797+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:54.898 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:54.899+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:54.932 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:24:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:24:54.932 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:24:54.932 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:54.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:24:55.000 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:55.000+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:55.017 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:08.005 42879== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:55.076 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:55.076 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:55.076 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:24:55.077 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:55.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:24:55.130 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:55.129+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:55.165 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:24:55.223 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:55.223 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:55.223 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:24:55.223 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:24:55.223 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:55.224 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:55.224+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:55.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:24:55.253 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:08.241 42879== Warning: unimplemented fcntl command: 1036 2024-06-03T05:24:55.302 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:55.303+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:55.354 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:24:55.403 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:55.404+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:55.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:24:55.472 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:55.472 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:24:55.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:24:55.472 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:24:55.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.472 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:24:55.472 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:24:55.472 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.587101337999997, 2024-06-03T05:24:55.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:24:55.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:24:55.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.477 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.412145006000003, 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:24:55.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:24:55.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.411182572000001, 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:24:55.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:24:55.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.410392569000003, 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:24:55.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.375738708, 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:24:55.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.375049210999997, 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:24:55.499 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:24:55.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.504 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.374375862000001, 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:24:55.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:24:55.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.301942746999998, 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:24:55.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:24:55.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.543955554999997, 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:24:55.515 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:24:55.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:24:55.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.521 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.295305233999997, 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:24:55.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.294059847, 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:24:55.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:24:55.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.293000337999999, 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:24:55.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:24:55.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:24:55.538 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.260151942999997, 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:24:55.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:24:55.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.543 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.259066812, 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:24:55.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:24:55.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.258049794000001, 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.549 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:24:55.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:24:55.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:24:55.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:24:55.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:24:55.555 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.238266277000001, 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:24:55.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:24:55.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:24:55.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.113658284000003, 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:24:55.562 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:24:55.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:24:55.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.112838619999998, 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:24:55.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:24:55.573 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.112137670999999, 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:24:55.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:24:55.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.705083938999998, 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:24:55.579 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:24:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:24:55.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:24:55.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:24:55.583 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:55.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:24:55.585 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:55.585+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:55.660 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:24:55.660 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:24:55.686 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:55.686+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:55.755 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:24:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:24:55.760 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:55.760 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:24:55.760 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:55.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:24:55.811 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:55.810+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:55.849 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:24:55.850 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:24:55.887 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:55.887+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:55.942 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:55.942 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:55.942 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:24:55.942 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:24:55.942 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:55.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:24:55.950 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:24:55.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:24:55.988 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:55.988+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:56.046 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:24:56.046 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:24:56.089 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:56.089+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:56.133 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:56.133 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:24:56.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:24:56.134 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:24:56.134 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:56.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:24:56.145 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:24:56.145 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:24:56.230 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:56.230+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:56.237 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:24:56.237 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:24:56.290 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:56.290+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:56.370 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:24:56.371 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:24:56.371 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:56.372 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:56.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:24:56.424 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:56.423+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:56.494 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:56.494+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:56.563 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:56.563 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:56.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:24:56.592 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:56.592+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:56.661 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:24:56.693 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:56.693+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:56.716 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:56.716 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:56.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:24:56.794 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:56.794+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:56.863 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:56.864 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:56.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:24:56.895 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:56.895+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:56.996 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:56.996+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:57.006 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:57.006 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:57.016 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:24:57.101 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:57.101+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:57.191 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:57.192 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:57.197 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:57.198+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:57.200 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:24:57.299 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:57.299+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:57.374 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:57.374 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:57.384 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:24:57.399 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:57.399+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:57.500 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:57.500+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:57.535 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:57.536 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:57.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:24:57.630 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:57.630+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:57.702 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:57.702+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:57.744 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:57.744 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:57.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:24:57.804 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:57.803+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:57.904 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:57.904+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:57.933 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:57.934 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:57.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:24:58.004 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:58.004+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:58.106 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:58.105+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:58.154 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:58.154 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:58.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:24:58.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:24:58.206 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:58.206+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:58.307 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:24:58.307+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:24:58.319 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:58.320 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:58.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:24:58.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:24:58.427 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:58.427+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:58.466 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:58.467 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:58.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:24:58.509 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:58.509+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:58.609 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:24:58.610+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:24:58.648 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:24:58.649 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:24:58.667 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:58.667 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:58.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:24:58.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:24:58.678 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:24:58.679 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:24:58.679 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.9170482534597207 2024-06-03T05:24:58.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 0.9170482534597207 2024-06-03T05:24:58.710 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:58.710+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:58.754 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:24:58.754 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:24:58.810 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:24:58.810+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:24:58.905 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:24:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:24:58.911 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:58.911+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:58.950 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:24:58.950 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:24:58.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:24:59.012 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:24:59.012+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:24:59.024 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:24:59.025 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:24:59.113 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:59.113+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:59.160 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:24:59.160 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:24:59.161 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:59.161 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:59.161 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:24:59.162 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:59.164 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:24:59.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:24:59.213 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:59.214+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:59.258 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:24:59.258 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:24:59.315 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:59.315+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:59.379 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:24:59.379 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:24:59.379 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:59.381 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:24:59.382 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:24:59.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:24:59.432 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:59.431+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:59.478 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:24:59.478 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:24:59.517 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:24:59.517+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:24:59.573 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:24:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:24:59.573 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:24:59.574 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:59.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:24:59.618 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:24:59.618+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:24:59.720 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:59.719+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:59.759 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:59.759 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:24:59.759 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:59.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:24:59.820 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:24:59.820+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:24:59.929 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:24:59.930+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:24:59.931 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:24:59.944 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:24:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:24:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:24:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:24:59.945 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:24:59.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:00.021 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:00.022+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:00.031 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 0.917048 (eac3) 2024-06-03T05:25:00.122 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:00.122+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:00.189 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.319728517000001, 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.144772185000001, 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:25:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:25:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:25:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.143809750999999, 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:25:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.143019748, 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:25:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.108365886999998, 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:25:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:25:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.107676390000002, 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:25:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.107003040999999, 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.034569926000003, 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:25:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.276582734000002, 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:25:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.027932413000002, 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:25:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:25:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.026687025999998, 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:25:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.025627516999997, 2024-06-03T05:25:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:25:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:25:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.992779122000002, 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:25:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.991693990999998, 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:25:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:25:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.990676972999999, 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.970893455999999, 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:25:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.846285463000001, 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:25:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.845465799000003, 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:25:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.844764849999997, 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:25:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:25:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.437711118000003, 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:25:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:25:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:25:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:00.297 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:00.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:25:00.299 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:00.299+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:00.399 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:00.399+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:00.473 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:00.473 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:00.473 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:00.473 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:00.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:25:00.499 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:00.500+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:00.600 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:00.600+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:00.616 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:00.616 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:00.616 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:00.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:25:00.710 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:00.710+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:00.799 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:25:00.799 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:00.802 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:00.802+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:00.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:00.902 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:00.903+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:00.981 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:00.981 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:00.981 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:00.981 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:00.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:01.035 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:01.035+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:01.104 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:01.104+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:01.126 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:01.126 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:01.126 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:01.126 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:01.127 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:01.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:01.220 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:01.220+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:01.306 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:01.306+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:01.325 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:01.325 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:01.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:01.325 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:01.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:01.325 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:01.325 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:01.325 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.455746279000003, 2024-06-03T05:25:01.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:01.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:01.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:25:01.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:01.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:01.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:01.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:01.330 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:01.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:01.407 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:01.407+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:01.481 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:01.481 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:01.481 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:01.481 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:01.490 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:01.490 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:25:01.508 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:01.508+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:01.609 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:01.609+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:01.686 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:01.686 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:01.686 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:01.686 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:01.686 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:01.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:01.710 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:01.710+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:01.791 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:25:01.791 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:25:01.811 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:01.811+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:01.831 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:01.831 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:01.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:01.891 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:25:01.892 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:25:01.912 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:01.912+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:01.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:25:01.988 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:01.988 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:01.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:02.001 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:25:02.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:25:02.046 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:02.046+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:02.114 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:02.114+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:02.143 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:25:02.144 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:25:02.150 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:02.150 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:02.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:02.181 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:02.239 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:25:02.239 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:25:02.240 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:02.240+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:02.315 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:02.315+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:02.334 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:25:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:25:02.338 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:02.338 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:02.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:02.444 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:25:02.445 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:25:02.445 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:02.445+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:02.517 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:02.516+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:02.538 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:25:02.538 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:25:02.567 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:02.568 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:02.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:02.617 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:02.617+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:02.724 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:02.721+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:02.767 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:02.767 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:02.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:02.819 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:02.818+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:02.920 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:02.919+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:02.967 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:02.967 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:02.976 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:03.031 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:03.032 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:03.032+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:03.047 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-06-03T05:25:03.047 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-03T05:25:03.047 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:25:03.103 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:25:03.090+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35004) UID: 0 2024-06-03T05:25:03.105 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:25:03.092+0000 9f9e640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-06-03T05:25:03.121 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:03.122+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:03.162 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:03.162 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:03.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:03.257 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:03.256+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:03.323 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:03.323+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:03.349 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:03.349 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:03.359 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:03.444 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:03.443+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:03.524 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:03.524+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:03.535 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:03.535 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:03.544 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:03.612 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:10.411 35037== 2024-06-03T05:25:03.612 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:10.411 35037== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:25:03.622 INFO:tasks.ceph.mon.c.smithi038.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:25:03.625 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:03.625+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:03.679 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:03.679 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:03.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:03.731 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:03.730+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:03.827 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:03.826+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:03.861 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:03.862 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:03.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:03.927 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:03.927+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:04.007 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:04.007 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:04.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:04.060 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:04.060+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:04.142 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:04.139+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:04.156 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:04.156 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:04.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:04.249 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:04.249+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:04.331 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:04.331+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:04.337 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:04.337 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:04.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:04.432 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:04.432+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:04.519 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:04.519 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:04.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:25:04.532 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:04.532+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:04.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:25:04.634 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:04.634+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:04.700 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:04.700 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:04.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:25:04.735 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:04.734+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:04.836 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:04.835+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:04.883 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:04.883 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:04.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:25:04.937 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:04.937+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:05.024 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:05.024 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:05.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:05.047 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:25:05.048 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:25:05.048 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-03T05:25:05.048 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-06-03T05:25:05.075 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:05.075+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:05.138 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:05.137+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:05.208 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:05.208 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:05.218 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:05.263 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:05.263+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:05.340 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:05.340+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:05.359 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:05.359 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:05.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:05.452 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:05.452+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:05.541 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:05.541 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:05.541 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:05.540+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:05.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:05.642 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:05.642+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:05.685 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:05.685 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:05.694 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:05.743 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:05.742+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:05.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-06-03T05:25:05.832 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:05.832 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:05.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:05.884 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:05.882+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:05.945 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:05.944+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:06.014 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:06.014 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:06.024 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:06.046 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:06.046+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:06.147 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:06.147+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:06.197 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:06.197 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:06.206 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:06.251 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:06.250+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:06.341 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:06.341 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:06.348 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:06.348+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:06.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:06.449 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:06.448+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:06.527 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:06.528 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:06.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:06.579 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:06.577+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:06.651 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:06.651+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:06.711 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:06.711 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:06.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:06.752 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:06.752+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:06.855 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:06.854+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:06.894 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:06.894 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:06.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:06.988 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:06.987+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:07.057 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:07.056+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:07.078 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:07.078 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:07.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:07.158 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:07.158+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:07.258 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:07.258+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:07.261 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:07.262 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:07.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:07.360 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:07.360+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:07.406 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:07.406 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:07.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:07.460 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:07.459+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:07.552 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:07.552 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:07.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:07.561 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:07.561+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:07.661 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:07.660+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:07.710 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:07.710 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:07.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:07.762 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:07.761+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:07.855 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:07.855 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:07.863 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:07.864+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:07.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:07.965 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:07.965+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:08.039 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:08.039 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:08.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:08.092 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:08.092+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:08.167 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:08.166+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:08.184 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:08.184 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:08.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:08.268 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:08.267+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:08.366 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:08.366 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:08.369 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:08.368+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:08.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:08.493 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:08.493+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:08.507 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:08.507 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:08.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:25:08.570 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:08.570+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:08.672 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:08.671+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:08.693 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:08.693 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:08.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:25:08.772 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:08.772+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:08.833 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:08.833 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:08.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:25:08.885 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:08.885+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:08.973 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:08.973 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:08.974 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:08.974+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:08.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:09.074 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:09.074+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:25:09.148 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-06-03T05:25:09.149 INFO:tasks.ceph.mon.c:Stopped 2024-06-03T05:25:09.149 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-06-03T05:25:09.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:09.156 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:09.156 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:09.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:09.209 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:09.209+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:09.276 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:09.275+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:09.299 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:09.300 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:09.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:09.394 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:09.394+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:09.478 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:09.478+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:09.483 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:09.483 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:09.492 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:09.579 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:09.579+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:09.634 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:09.634 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:09.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:09.680 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:09.680+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:09.776 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:09.777 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:09.780 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:09.780+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:09.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:09.882 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:09.881+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:09.920 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:09.920 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:09.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:10.014 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:10.013+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:10.085 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:10.084+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:10.102 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:10.102 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:10.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:10.184 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:10.185+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:10.285 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:10.285 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:10.285 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:10.285+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:10.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:10.386 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:10.387+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:10.469 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:10.469 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:10.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:10.522 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:10.521+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:10.589 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:10.588+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:10.611 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:10.611 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:10.620 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:10.690 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:10.690+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:10.791 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:10.791+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:10.796 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:10.797 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:10.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:10.892 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:10.891+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:10.940 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:10.941 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:10.949 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:10.992 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:10.992+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:11.108 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:11.106+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:11.127 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:11.127 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:11.136 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:11.194 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:11.193+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:11.270 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:11.271 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:11.279 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:11.322 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:11.320+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:11.396 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:11.396+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:11.460 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:11.460 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:11.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:11.514 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:11.514+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:11.597 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:11.597+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:11.614 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:11.614 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:11.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:11.709 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:11.709+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:11.798 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:11.798+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:11.799 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:11.799 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:11.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:11.899 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:11.898+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:11.982 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:11.982 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:11.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:12.000 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:12.000+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:12.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-06-03T05:25:12.100 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:12.100+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:12.123 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:12.123 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:12.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:12.201 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:12.201+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:12.265 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:12.265 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:12.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:12.302 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:12.302+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:12.403 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:12.403+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:12.470 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:12.470 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:12.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:25:12.523 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:12.523+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:12.606 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:12.606+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:12.614 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:12.614 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:12.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:25:12.708 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:12.708+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:12.797 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:12.797 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:12.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:25:12.807 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:12.807+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:12.868 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:12.923 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:12.922+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:12.980 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:12.980 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:12.989 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:13.033 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:13.033+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:13.110 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:13.109+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:13.124 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:13.124 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:13.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:13.218 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:13.218+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:13.307 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:13.308 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:13.311 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:13.311+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:13.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:13.413 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:13.412+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:13.490 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:13.490 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:13.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:13.544 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:13.543+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:13.615 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:13.614+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:13.636 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:13.636 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:13.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:13.716 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:13.716+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:13.817 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:13.817+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:13.820 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:13.820 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:13.829 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:13.918 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:13.918+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:13.962 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:13.963 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:13.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:14.019 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:14.019+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:14.106 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:14.106 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:14.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:14.119 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:14.120+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:14.220 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:14.221+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:14.257 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:14.257 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:14.266 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:14.321 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:14.321+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:14.398 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:14.398 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:14.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:14.452 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:14.451+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:14.524 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:14.524+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:14.542 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:14.542 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:14.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:14.624 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:14.625+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:14.726 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:14.725+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:14.728 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:14.729 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:14.738 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:14.827 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:14.826+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:14.911 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:14.911 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:14.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:14.928 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:14.928+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:15.031 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:15.031+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:15.096 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:15.096 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:15.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:15.131 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:15.131+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:15.232 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:15.231+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:15.241 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:15.241 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:15.250 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:15.333 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:15.334+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:15.430 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:15.430 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:15.434 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:15.434+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:15.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:15.536 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:15.535+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:15.587 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:15.587 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:15.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:15.640 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:15.640+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:15.730 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:15.731 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:15.736 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:15.736+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:15.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:15.837 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:15.837+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:15.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:15.927 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:15.927 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:15.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:15.937 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:15.937+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:16.039 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:16.038+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:16.110 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:16.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:16.165 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:16.165+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:16.240 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:16.240+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:16.252 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:16.252 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:16.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:16.342 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:16.341+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:16.442 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:16.442 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:16.443 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:16.443+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:16.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:25:16.544 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:16.544+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:16.584 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:16.584 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:16.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:25:16.645 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:16.645+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:16.746 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:16.746+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:16.767 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:16.768 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:16.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:25:16.846 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:16.846+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:16.948 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:16.947+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:16.949 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:16.949 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:16.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:17.048 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:17.048+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:17.093 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:17.094 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:17.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:17.149 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:17.149+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:17.233 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:17.233 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:17.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:17.249 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:17.249+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:17.351 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:17.350+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:17.379 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:17.379 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:17.389 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:17.452 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:17.452+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:17.540 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:17.540 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:17.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:17.552 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:17.552+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:17.673 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:17.672+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:17.687 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:17.687 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:17.696 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:17.754 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:17.754+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:17.855 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:17.854+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:17.869 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:17.869 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:17.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:17.956 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:17.956+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:18.052 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:18.052 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:18.057 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:18.057+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:18.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:18.158 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:18.158+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:18.199 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:18.199 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:18.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:18.259 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:18.259+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:25:18.318 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 31, 2024-06-03T05:25:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "ebc09f7e-f094-4421-826f-479fc526fc62", 2024-06-03T05:25:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-06-03T05:23:10.198378+0000", 2024-06-03T05:25:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-06-03T05:25:03.492231+0000", 2024-06-03T05:25:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-06-03T05:24:09.927340+0000", 2024-06-03T05:25:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-06-03T05:23:16.043190+0000", 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5804032, 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 6, 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-06-03T05:25:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 2, 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "jewel", 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-06-03T05:23:56.753149+0000", 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-06-03T05:25:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 1, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-06-03T05:25:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "16", 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-03T05:25:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-06-03T05:25:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-06-03T05:25:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-06-03T05:25:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 8, 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 8, 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.25, 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2, 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2, 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-06-03T05:25:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "rbd", 2024-06-03T05:25:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-06-03T05:24:12.898324+0000", 2024-06-03T05:25:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-06-03T05:25:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-06-03T05:25:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-06-03T05:25:18.329 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-06-03T05:25:18.329 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 1, 2024-06-03T05:25:18.329 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-06-03T05:25:18.329 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-06-03T05:25:18.329 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-06-03T05:25:18.329 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 8, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 8, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 8, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 8, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 8, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-06-03T05:25:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "24", 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-06-03T05:25:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-06-03T05:25:18.332 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-06-03T05:25:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 4.9800000190734863, 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 4.9800000190734863, 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 4.380000114440918, 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 4.380000114440918, 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-06-03T05:25:18.334 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "32b256d9-eee7-435b-8aec-fef2ba90366c", 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 16, 2024-06-03T05:25:18.335 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 0, 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.337 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/905141127", 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6811/905141127", 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/905141127", 2024-06-03T05:25:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6813/905141127", 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "371c2e00-54db-430b-8497-a4c9ff6753b9", 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0.9170379638671875, 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:25:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 18, 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.340 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:25:18.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6810/3490615740", 2024-06-03T05:25:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6812/3490615740", 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6815/3490615740", 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6814/3490615740", 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "e8145335-ec5d-4dd1-9250-a4ab5b159d29", 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:25:18.343 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 0, 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.345 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.346 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/3359326199", 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/3359326199", 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/3359326199", 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/3359326199", 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c", 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:25:18.347 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 19, 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-06-03T05:25:18.348 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:25:18.349 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:25:18.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/2289724788", 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/2289724788", 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/2289724788", 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/2289724788", 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.351 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "46b46f54-04d0-46c0-809c-64fa385e6780", 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.352 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6808", 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6809", 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6811", 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.354 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6810", 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6808/3833397294", 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6809/3833397294", 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6811/3833397294", 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6810/3833397294", 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f7a50ac0-eba9-4602-b90a-2dbc4bb20593", 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:25:18.356 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6800", 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6801", 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.357 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6803", 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.358 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6802", 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6800/3915827095", 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6801/3915827095", 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6803/3915827095", 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6802/3915827095", 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "081024c0-2482-4846-a7bd-594a58f2ceef", 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 0, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:25:18.360 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6804", 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6805", 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.361 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6807", 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.362 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6806", 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6804/1144336651", 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6805/1144336651", 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1144336651", 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6806/1144336651", 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "6234c757-8f7b-497e-9347-4a7983df592c", 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:25:18.364 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6812", 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6813", 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:25:18.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6815", 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:25:18.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6814", 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6812/370645108", 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6813/370645108", 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6815/370645108", 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6814/370645108", 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:18.367 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:56.081253+0000", 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.368 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:55.642638+0000", 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:25:18.369 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:55.131979+0000", 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:55.147738+0000", 2024-06-03T05:25:18.370 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.907705+0000", 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:25:18.371 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.533669+0000", 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.310102+0000", 2024-06-03T05:25:18.372 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.959957+0000", 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-06-03T05:25:18.373 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [], 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [], 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": {}, 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:18.375 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:18.377 INFO:tasks.thrashosds.thrasher:j is {'epoch': 31, 'fsid': 'ebc09f7e-f094-4421-826f-479fc526fc62', 'created': '2024-06-03T05:23:10.198378+0000', 'modified': '2024-06-03T05:25:03.492231+0000', 'last_up_change': '2024-06-03T05:24:09.927340+0000', 'last_in_change': '2024-06-03T05:23:16.043190+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-03T05:23:56.753149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-06-03T05:24:12.898324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '32b256d9-eee7-435b-8aec-fef2ba90366c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6809', 'nonce': 905141127}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6811', 'nonce': 905141127}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6816', 'nonce': 905141127}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6813', 'nonce': 905141127}]}, 'public_addr': '172.21.15.38:6809/905141127', 'cluster_addr': '172.21.15.38:6811/905141127', 'heartbeat_back_addr': '172.21.15.38:6816/905141127', 'heartbeat_front_addr': '172.21.15.38:6813/905141127', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '371c2e00-54db-430b-8497-a4c9ff6753b9', 'up': 1, 'in': 1, 'weight': 0.9170379638671875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6810', 'nonce': 3490615740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6812', 'nonce': 3490615740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6815', 'nonce': 3490615740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6814', 'nonce': 3490615740}]}, 'public_addr': '172.21.15.38:6810/3490615740', 'cluster_addr': '172.21.15.38:6812/3490615740', 'heartbeat_back_addr': '172.21.15.38:6815/3490615740', 'heartbeat_front_addr': '172.21.15.38:6814/3490615740', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8145335-ec5d-4dd1-9250-a4ab5b159d29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6805', 'nonce': 3359326199}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6806', 'nonce': 3359326199}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6808', 'nonce': 3359326199}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6807', 'nonce': 3359326199}]}, 'public_addr': '172.21.15.38:6805/3359326199', 'cluster_addr': '172.21.15.38:6806/3359326199', 'heartbeat_back_addr': '172.21.15.38:6808/3359326199', 'heartbeat_front_addr': '172.21.15.38:6807/3359326199', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6801', 'nonce': 2289724788}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6802', 'nonce': 2289724788}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6804', 'nonce': 2289724788}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6803', 'nonce': 2289724788}]}, 'public_addr': '172.21.15.38:6801/2289724788', 'cluster_addr': '172.21.15.38:6802/2289724788', 'heartbeat_back_addr': '172.21.15.38:6804/2289724788', 'heartbeat_front_addr': '172.21.15.38:6803/2289724788', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46b46f54-04d0-46c0-809c-64fa385e6780', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 3833397294}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6809', 'nonce': 3833397294}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6811', 'nonce': 3833397294}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 3833397294}]}, 'public_addr': '172.21.15.64:6808/3833397294', 'cluster_addr': '172.21.15.64:6809/3833397294', 'heartbeat_back_addr': '172.21.15.64:6811/3833397294', 'heartbeat_front_addr': '172.21.15.64:6810/3833397294', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f7a50ac0-eba9-4602-b90a-2dbc4bb20593', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 3915827095}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6801', 'nonce': 3915827095}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6803', 'nonce': 3915827095}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 3915827095}]}, 'public_addr': '172.21.15.64:6800/3915827095', 'cluster_addr': '172.21.15.64:6801/3915827095', 'heartbeat_back_addr': '172.21.15.64:6803/3915827095', 'heartbeat_front_addr': '172.21.15.64:6802/3915827095', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '081024c0-2482-4846-a7bd-594a58f2ceef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1144336651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6805', 'nonce': 1144336651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6807', 'nonce': 1144336651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1144336651}]}, 'public_addr': '172.21.15.64:6804/1144336651', 'cluster_addr': '172.21.15.64:6805/1144336651', 'heartbeat_back_addr': '172.21.15.64:6807/1144336651', 'heartbeat_front_addr': '172.21.15.64:6806/1144336651', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6234c757-8f7b-497e-9347-4a7983df592c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 370645108}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6813', 'nonce': 370645108}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6815', 'nonce': 370645108}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 370645108}]}, 'public_addr': '172.21.15.64:6812/370645108', 'cluster_addr': '172.21.15.64:6813/370645108', 'heartbeat_back_addr': '172.21.15.64:6815/370645108', 'heartbeat_front_addr': '172.21.15.64:6814/370645108', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:56.081253+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:55.642638+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:55.131979+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:55.147738+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.907705+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.533669+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.310102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.959957+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-03T05:25:18.377 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-06-03T05:25:18.378 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:18.378 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:18.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:18.379 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:18.377+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:18.426 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:25:18.427 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:25:18.477 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:18.477+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:18.514 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:18.514 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:18.524 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:18.526 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:25:18.526 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:25:18.578 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:18.578+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:18.629 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:25:18.629 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:25:18.679 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:18.679+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:18.692 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:18.692 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:18.702 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:18.733 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:25:18.733 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:25:18.780 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:18.780+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:18.841 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:25:18.841 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:25:18.859 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:18.859 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:18.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:18.860 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:25:18.860 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:18.868 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:18.909 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:18.909+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:18.934 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:25:18.934 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:25:18.983 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:18.983+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:19.027 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:25:19.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:25:19.043 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:19.043 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:19.043 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:19.044 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:19.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:19.095 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:19.095+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:19.120 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:25:19.120 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:25:19.184 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:19.184+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:19.260 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:19.260 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:19.260 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:19.260 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:19.260 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:19.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:19.284 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:19.284+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:19.385 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:19.385+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:19.426 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:19.426 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:19.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:19.426 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:25:19.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:19.426 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:19.426 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:19.426 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.759768645999998, 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:19.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:25:19.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:25:19.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:25:19.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:19.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:19.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:19.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:25:19.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:25:19.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:19.433 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:19.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:19.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:19.433 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:19.433 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:19.433 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:25:19.436 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:19.440 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:19.451 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-06-03T05:25:19.452 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-03T05:25:19.452 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-06-03T05:25:19.492 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:19.492+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:19.494 INFO:tasks.ceph.mon.c:Started 2024-06-03T05:25:19.494 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-06-03T05:25:19.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:19.577 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:19.577 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:19.577 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:19.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:19.588 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:19.587+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:19.688 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:19.688+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:19.722 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:19.722 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:19.722 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:19.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:19.789 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:19.789+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:19.878 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:19.878 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:25:19.878 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:19.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:19.889 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:19.889+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:19.991 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:19.990+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:20.063 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:20.063 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:20.064 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:20.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:20.091 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:20.091+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:20.192 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:20.192+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:20.205 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:20.205 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:20.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:20.254 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:20.293 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:20.293+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:20.394 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:20.394+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:20.397 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:20.398 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:20.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:20.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:20.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.398 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:25:20.398 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:20.398 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 81.528450054000004, 2024-06-03T05:25:20.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:25:20.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:25:20.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.353493721999996, 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:25:20.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:25:20.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.352531287999994, 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.351741285000003, 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:25:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.317087423999993, 2024-06-03T05:25:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:25:20.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.424 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.316397926999997, 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:25:20.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:25:20.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:25:20.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:25:20.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:25:20.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:20.430 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.315724578000001, 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:25:20.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.435 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.243291463000006, 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:20.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:25:20.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:25:20.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:25:20.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:25:20.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:25:20.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:25:20.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.485304271000004, 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.442 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:20.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:25:20.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:25:20.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:25:20.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:25:20.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.447 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.236653950000004, 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:25:20.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:25:20.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:25:20.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:25:20.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:25:20.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:25:20.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:25:20.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:25:20.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:25:20.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:25:20.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:25:20.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.455 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.455 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:20.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.235408562999993, 2024-06-03T05:25:20.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:25:20.456 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.456 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.456 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:25:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:25:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:25:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:25:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:25:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.234349054000006, 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:25:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:25:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.201500659000004, 2024-06-03T05:25:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:25:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:25:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.200415527999994, 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.199398509999995, 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:25:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.179614993000001, 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:25:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.055007000000003, 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:25:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.054187335999998, 2024-06-03T05:25:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:25:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:25:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:25:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:25:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:25:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:25:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:25:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:25:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:25:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:25:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:25:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:25:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:25:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:25:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:25:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:25:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:25:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:25:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:25:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:25:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.053486387000007, 2024-06-03T05:25:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:25:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:25:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:25:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:25:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:25:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:25:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:25:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:25:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:25:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:25:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:25:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:25:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:25:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:25:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:25:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:25:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:25:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:25:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:25:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:25:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:25:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.646432654999998, 2024-06-03T05:25:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:25:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:25:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:25:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:25:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:25:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:25:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:25:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:25:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:25:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:25:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:25:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:25:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:25:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:20.532 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:20.534 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:25:20.535 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:20.534+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:20.635 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:20.635+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:20.708 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:20.709 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:20.709 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:20.709 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:20.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:25:20.760 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:20.759+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:20.836 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:20.836+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:20.892 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:20.893 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:20.902 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:25:20.937 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:20.937+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:21.034 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:25:21.035 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:21.038 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:21.038+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:21.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:21.134 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:25:21.179 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:21.179+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:21.180 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:21.180 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:21.180 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:21.180 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:21.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:21.239 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:21.239+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:21.325 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:21.325 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:21.325 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:21.325 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:21.325 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:21.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:21.339 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:21.340+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:21.401 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:25:21.402 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:25:21.440 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:21.440+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:21.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:25:21.498 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:25:21.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:25:21.524 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 62.653187946000003, 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:21.529 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:21.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:21.579 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:21.579+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:21.601 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:25:21.601 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:25:21.641 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:21.641+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:21.682 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:21.683 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:21.692 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:21.734 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:25:21.734 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:25:21.742 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:21.742+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:21.830 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:25:21.831 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:25:21.843 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:21.844+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:21.866 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:21.866 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:21.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:21.935 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:25:21.936 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:25:21.944 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:21.945+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:22.033 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:25:22.034 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:25:22.045 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:22.045+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:22.053 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:22.053 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:22.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:22.147 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:22.147+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:22.238 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:22.238 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:22.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:22.248 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:22.248+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:22.332 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:25:22.333 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:25:22.349 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:22.349+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:22.383 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:22.383 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:22.392 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:22.449 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:22.450+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:22.540 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:35.528 42879== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:22.550 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:22.550+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:22.567 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:22.567 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:22.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:22.595 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:29.340 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:22.597 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:29.341 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:22.601 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:35.589 42879== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:22.621 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:29.366 34626== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:22.651 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:22.651+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:22.712 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:22.712 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:22.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:22.752 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:22.752+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:22.853 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:22.853+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:22.903 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:22.903 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:22.911 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:35.899 42879== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:22.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:22.957 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:22.956+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:23.054 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:23.054+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:23.092 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:23.093 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:23.093 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:25:23.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:23.155 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:23.155+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:23.256 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:23.256+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:23.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:23.279 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:23.280 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:23.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:23.356 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:23.357+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:23.378 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:25:23.378 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:25:23.378 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-06-03T05:25:23.378 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-03T05:25:23.378 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-03T05:25:23.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-06-03T05:25:23.458 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:23.458+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:23.474 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:23.474 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:23.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:23.515 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:23.516 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-03T05:25:23.516 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:23.559 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:23.559+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:23.635 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:23.636 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:23.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:23.689 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:23.689+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:23.761 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:23.761+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:23.779 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:23.779 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:23.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:23.873 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:23.873+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:23.962 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:23.962 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:23.962 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:23.963+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:23.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:23.978 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:24.064 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:24.064+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:24.147 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:24.147 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:24.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:24.200 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:24.200+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:24.266 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:24.266+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:24.289 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:24.289 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:24.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:24.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:25:24.391 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:24.391+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:24.468 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:24.468+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:24.475 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:24.475 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:24.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:24.572 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:24.572+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:24.593 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:25:24.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:25:24.661 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:24.661 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:24.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:25:24.670 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:24.670+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:24.695 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:25:24.695 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:25:24.771 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:24.771+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:24.789 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:25:24.789 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:25:24.843 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:24.843 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:24.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:25:24.884 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:25:24.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:25:24.885 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:24.884+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:24.972 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:24.972+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:25.018 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:25:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:25:25.063 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:25.064 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:25.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:25:25.074 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:25.073+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:25.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:25:25.115 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:25:25.115 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:25:25.175 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:25.175+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:25.218 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:25:25.219 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:25:25.261 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:25.262 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:25.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:25.262 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:25:25.262 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:25.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:25.315 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:25.315+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:25.316 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:25:25.316 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:25:25.391 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:25.391+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:25.406 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:25.406 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:25.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:25.477 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:25.478+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:25.578 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:25.579+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:25.588 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:25.588 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:25.589 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:25.589 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:25.589 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:25.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:25.683 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:25.683+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:25.776 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 66.906112277000005, 2024-06-03T05:25:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:25.780 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:25.782 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:25.782+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:25.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:25.882 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:25.883+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:25.937 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:25.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:25.938 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:25.938 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:25.947 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:25.983 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:25.984+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:26.084 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:26.084+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:26.137 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:26.137 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:26.137 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:26.137 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:26.137 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:26.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:26.190 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:26.190+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:26.248 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:25:26.304 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:26.304+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:26.311 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:25:26.312 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:26.322 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:26.407 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:26.407+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:26.488 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:26.489+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:26.496 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:26.496 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:26.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:26.589 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:26.590+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:26.681 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:26.681 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:26.690 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:26.691 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:26.690+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:26.790 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:26.791+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:26.875 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:25:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.053359853000003, 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:25:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 74.794528049999997, 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:25:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:25:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:26.884 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:26.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:26.927 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:26.927+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:26.991 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:26.992+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:26.993 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:27.067 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:27.068 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:27.068 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:27.068 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:27.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:27.092 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:27.092+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:27.193 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:27.193+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:27.216 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:27.216 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:27.216 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:27.216 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:27.216 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:27.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:27.310 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:27.310+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:27.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:25:27.394 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:27.394+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:27.405 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:27.405 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:27.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:27.405 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:25:27.405 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:27.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:27.495 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:27.496+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:27.542 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:07.987 46030== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:27.592 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:27.593 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:27.593 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:27.593 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:27.597 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:27.597+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:27.602 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:27.698 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:27.698+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:27.750 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:27.753 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:25:27.754 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:25:27.786 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:27.787 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:27.787 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:27.787 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:27.787 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:27.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:27.798 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:27.799+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:27.804 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:08.249 46030== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:27.883 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:25:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:25:27.899 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:27.899+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.278398585999994, 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:25:27.943 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:27.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:25:27.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:27.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:27.947 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:27.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:27.997 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:25:27.998 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:25:28.000 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:28.000+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:28.087 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:28.087 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:28.087 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:28.087 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:28.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:28.111 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:25:28.112 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:25:28.140 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:28.140+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:28.202 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:28.203+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:28.216 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:25:28.216 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:25:28.258 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:28.258 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:28.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:25:28.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:28.308 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:28.308+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:28.316 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:25:28.316 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:25:28.405 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:28.405+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:28.428 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:25:28.428 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:25:28.442 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:28.442 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:28.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:28.506 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:28.506+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:28.586 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:25:28.586 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:25:28.586 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-06-03T05:25:28.586 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-03T05:25:28.586 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-03T05:25:28.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-06-03T05:25:28.588 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:25:28.589 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:25:28.623 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:28.623 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:28.631 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:28.631+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:28.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:28.717 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:28.717+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:28.727 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:28.727 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-03T05:25:28.727 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:28.808 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:28.808+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:28.820 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:28.820 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:28.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:28.909 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:28.909+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:28.962 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:28.962 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:28.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:25:29.016 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:29.016+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:29.104 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:29.104 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:29.111 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:29.111+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:29.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:25:29.212 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:29.212+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:29.288 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:29.289 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:29.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:25:29.313 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:29.313+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:29.414 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:29.414+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:29.430 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:29.430 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:29.440 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:29.515 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:29.516+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:29.573 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:29.573 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:29.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:29.616 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:29.616+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:29.715 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:29.715 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:29.717 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:29.717+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:29.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:29.817 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:29.818+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:29.867 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:29.867 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:29.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:29.921 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:29.921+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:29.964 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:25:30.020 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:30.020 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:30.021 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:30.021+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:30.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:30.120 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:30.121+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:30.221 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:30.221 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:30.222 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:30.222+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:30.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:30.323 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:30.323+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:30.418 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:30.418 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:30.424 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:30.424+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:30.428 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:30.525 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:30.525+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:30.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:25:30.601 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:30.601 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:30.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:30.626 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:30.626+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:30.727 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:30.727+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:30.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:30.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:30.827 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:30.827+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:30.872 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:25:30.872 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:25:30.929 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:30.929+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:30.978 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:25:30.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:25:31.026 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:31.026 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 79.209003886999994, 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:31.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:25:31.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:25:31.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 78.950172084000002, 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:25:31.031 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:25:31.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:31.035 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:31.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:31.037 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:31.037+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:31.086 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:25:31.087 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:25:31.146 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:31.146+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:31.218 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:25:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:25:31.237 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:31.238 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:31.245 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:31.247 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:31.247+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:31.312 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:25:31.312 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:25:31.340 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:31.340+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:31.405 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:25:31.405 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:25:31.426 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:31.426 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:31.426 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:31.427 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:31.427 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:31.438 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:31.476 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:31.476 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:31.477+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:31.512 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:25:31.512 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:25:31.542 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:31.542+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:31.554 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:11.999 46030== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:31.610 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:12.055 46030== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:31.620 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:25:31.621 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:25:31.623 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:31.623 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:31.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:31.623 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:25:31.624 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:31.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:31.680 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:31.680+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:31.743 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:31.743+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:31.816 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:31.816 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:31.816 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:31.817 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:31.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:31.843 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:31.843+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:31.861 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:12.306 46030== Warning: unimplemented fcntl command: 1036 2024-06-03T05:25:31.944 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:31.944+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:31.984 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:25:31.999 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:31.999 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:31.999 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:31.999 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:32.000 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:32.009 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:32.054 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:32.051+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:32.146 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:32.146+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:32.204 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:25:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 74.53978712, 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:25:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:32.209 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:32.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:32.258 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:32.259+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:32.347 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:32.347 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:32.347 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:32.347 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:32.348 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:32.348+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:32.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:32.449 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:32.449+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:32.532 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:32.532 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:32.532 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:32.532 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:32.532 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:32.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:32.549 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:32.550+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:32.651 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:32.651+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:32.715 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:32.715 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:32.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:32.716 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:25:32.716 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:32.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:32.751 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:32.752+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:32.852 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:32.853+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:32.858 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:32.859 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:32.859 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:32.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:32.954 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:32.954+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:33.041 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:33.041 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:33.041 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:33.041 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:33.041 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:33.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:33.094 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:33.094+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:33.158 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:33.158+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:33.243 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:33.243 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.374313775000005, 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:25:33.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:25:33.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:33.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.199357442999997, 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:25:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:25:33.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.198395008999995, 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:25:33.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:25:33.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:25:33.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.197605006000003, 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:25:33.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:25:33.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:25:33.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:25:33.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:33.268 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.162951144999994, 2024-06-03T05:25:33.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:25:33.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.269 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:25:33.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:25:33.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:25:33.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:25:33.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:25:33.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.274 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.274 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:33.277 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.162261647999998, 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.278 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:25:33.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:25:33.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.161588299000002, 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:33.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:25:33.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:25:33.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:25:33.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:25:33.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:25:33.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:25:33.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:25:33.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:25:33.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:25:33.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:25:33.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:25:33.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:25:33.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:25:33.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:25:33.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:25:33.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:25:33.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:25:33.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:25:33.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:33.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.089155184000006, 2024-06-03T05:25:33.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:25:33.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.294 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:25:33.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:25:33.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:25:33.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:25:33.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:25:33.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:25:33.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:25:33.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:25:33.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:25:33.299 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.299 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.331167992000005, 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:33.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:25:33.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.305 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.082517671000005, 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:25:33.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:25:33.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:25:33.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:25:33.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.311 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.081272283999994, 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:25:33.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:25:33.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:25:33.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:25:33.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:25:33.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.080212775000007, 2024-06-03T05:25:33.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:25:33.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:25:33.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:25:33.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:25:33.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:25:33.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:25:33.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:25:33.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:25:33.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:25:33.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.324 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.324 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.324 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:33.324 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.047364380000005, 2024-06-03T05:25:33.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:25:33.325 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.325 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.325 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.326 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:25:33.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.326 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:33.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:33.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:25:33.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:25:33.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.046279248999994, 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.045262230999995, 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:25:33.338 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:25:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:25:33.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.025478714000002, 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.344 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:25:33.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:25:33.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:25:33.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:25:33.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:25:33.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:25:33.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:25:33.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:25:33.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:25:33.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:25:33.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:25:33.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:25:33.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:25:33.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:25:33.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.351 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.352 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.352 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:33.352 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:33.352 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 87.900870721000004, 2024-06-03T05:25:33.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:25:33.353 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.353 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.353 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:25:33.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 87.900051056999999, 2024-06-03T05:25:33.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 87.899350107999993, 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:25:33.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:25:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 81.492296375999999, 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:33.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:33.372 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:33.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:25:33.374 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:33.374+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:33.475 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:33.475+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:33.547 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:33.547 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:33.547 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:33.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:25:33.576 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:33.576+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:33.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:25:33.677 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:33.677+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:33.730 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:33.730 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:33.731 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:33.731 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:33.731 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:33.737 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:25:33.737 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:25:33.737 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:25:33.738 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-06-03T05:25:33.738 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:25:33.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:25:33.746 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:33.739+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35176) UID: 0 2024-06-03T05:25:33.746 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:25:33.739+0000 b485640 -1 osd.6 35 *** Got signal Terminated *** 2024-06-03T05:25:33.790 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:33.790+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:33.878 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:33.878+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:33.879 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:25:33.879 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:33.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:33.979 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:33.979+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:34.026 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:34.026 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:34.026 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:34.026 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:34.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:34.082 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:34.080+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:34.184 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:34.184+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:34.209 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:34.210 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:34.220 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:34.285 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:34.285+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:25:34.353 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:34.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:34.386 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:34.386+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:34.491 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:34.492 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:34.491+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:34.500 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:34.501 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:34.509 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:34.589 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:34.589+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:34.688 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:34.688+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:34.693 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:34.693 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:34.702 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:34.788 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:34.789+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:34.881 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 83.063170798000002, 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:25:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:25:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:25:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:25:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:25:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:25:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 82.804338994999995, 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:25:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:25:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:25:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:25:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:25:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:25:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:34.892 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:34.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:34.894 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:34.894+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:34.994 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:34.995+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:35.067 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:35.067 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:35.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:35.117 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:35.117+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:35.198 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:35.198+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:35.253 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:35.253 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:35.262 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:35.299 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:35.299+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:35.395 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:25:35.395 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:35.399 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:35.399+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:35.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:35.500 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:35.500+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:35.540 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:35.540 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:35.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:35.600 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:35.601+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:35.702 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:35.702+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:35.726 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:35.726 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:35.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:35.803 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:35.803+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:35.873 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 78.209171488999999, 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:35.874 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:35.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:25:35.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:35.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:35.878 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:35.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:35.927 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:35.927+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:36.004 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:36.004+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:36.024 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:36.025 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:36.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:36.105 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:36.106+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:36.207 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:36.207+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:36.207 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:36.208 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:36.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:36.308 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:36.308+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:36.352 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:36.352 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:36.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:36.352 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:25:36.352 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:36.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:36.410 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:36.410+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:36.510 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:36.510+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:36.533 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:36.533 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:36.533 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:36.534 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:36.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:36.627 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:36.627+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:36.712 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:36.712+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:36.716 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:36.717 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:36.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:36.813 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:36.813+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:36.914 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:36.914+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:36.952 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:36.952 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:36.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:36.952 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:36.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.083055040999994, 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:25:36.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.957 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.908098709000001, 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:36.958 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:25:36.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:36.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.964 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:36.964 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:36.964 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.907136274999999, 2024-06-03T05:25:36.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:25:36.965 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:36.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:36.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:25:36.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:25:36.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:36.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:25:36.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:25:36.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:25:36.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.906346271999993, 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:25:36.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:36.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:36.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.871692410999998, 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:36.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:36.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:25:36.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:36.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:25:36.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:25:36.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:36.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:25:36.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:25:36.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:36.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:25:36.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:25:36.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:25:36.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:36.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.871002914000002, 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:36.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:25:36.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.987 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.870329565000006, 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:36.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:25:36.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.797896449999996, 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:36.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:36.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:36.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:25:36.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:25:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.039909257999994, 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:36.999 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:25:37.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:25:37.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:25:37.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:25:37.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.791258936999995, 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:25:37.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:25:37.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:25:37.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.009 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.790013549999998, 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.014 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.788954040999997, 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:25:37.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:25:37.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.019 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.756105645999995, 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.024 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.755020514999998, 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:25:37.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.754003496999999, 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:25:37.030 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:37.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:25:37.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.734219980000006, 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.036 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:25:37.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 91.609611986999994, 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 91.608792323000003, 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:25:37.046 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:25:37.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 91.608091373999997, 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:25:37.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:25:37.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:25:37.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.201037642000003, 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:25:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:25:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:25:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.061 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:37.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:37.063 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:37.063+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:37.164 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:37.164+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:37.236 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:37.236 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:37.236 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:37.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:37.287 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:37.286+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:37.366 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:37.366+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:37.422 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:37.422 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:37.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:37.466 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:37.467+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:37.568 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:37.568+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:37.613 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:37.613 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 78.744215432000004, 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:37.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:37.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:25:37.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:37.618 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:37.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:37.668 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:37.669+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:37.759 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:37.759 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:37.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:37.770 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:37.769+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:37.871 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:37.870+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:37.946 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:37.947 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:37.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:38.006 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:38.005+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:38.072 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:38.072+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:38.093 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:25:38.093 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:38.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:38.187 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:38.187+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:38.275 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:38.275+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:38.283 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:38.283 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:38.292 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:38.375 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:38.375+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:38.422 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:38.476 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:38.476+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:38.477 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:38.478 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:38.490 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:38.540 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:25:38.540 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:25:38.577 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:38.577+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:38.637 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:25:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:25:38.678 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:38.678+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:38.680 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:38.681 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:38.690 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:38.733 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:25:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:25:38.779 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:38.779+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:38.835 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:25:38.835 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:25:38.871 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:38.871 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:38.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:38.881 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:38.880+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:38.936 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:25:38.937 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:25:38.981 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:38.980+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:39.037 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:25:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:25:39.058 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:39.058 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:39.068 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:39.082 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:39.082+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:39.183 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:39.183+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:39.243 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:39.243 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:39.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:39.297 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:39.297+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:39.386 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:39.385+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:39.387 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:39.387 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:39.396 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:39.487 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:39.486+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:39.572 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:39.572 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:39.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:39.625 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:39.625+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:39.688 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:39.687+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:39.722 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:39.722 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:39.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:39.789 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:39.788+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:39.887 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:39.887 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:39.889 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:39.890+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:39.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:39.991 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:39.990+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:40.075 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:40.075 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:40.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:40.126 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:40.125+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:40.192 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:40.192+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:40.264 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:40.264 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:40.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:40.317 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:40.316+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:40.393 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:40.393+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:40.407 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:40.408 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:40.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:40.494 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:40.494+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:40.591 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:40.591 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:40.594 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:40.595+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:40.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:40.696 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:40.696+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:40.733 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:40.733 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:40.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:40.797 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:40.797+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:40.878 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:40.878 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:40.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:40.931 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:40.930+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:40.998 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:40.998+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:41.023 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:41.023 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:41.023 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:41.023 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:41.024 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:41.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:41.099 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:41.099+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:41.200 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:41.199+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 82.340732012999993, 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:25:41.210 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:41.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:25:41.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:41.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:41.214 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:41.220 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:41.302 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:41.301+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:41.370 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:41.371 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:41.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:41.423 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:41.422+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:41.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:41.503 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:41.502+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:41.555 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:41.555 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:41.565 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:41.609 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:41.609+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:41.697 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:41.697 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:41.705 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:41.704+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:41.706 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:41.806 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:41.806+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:41.884 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:41.884 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:41.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:41.935 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:41.935+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:42.007 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:42.008+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:42.067 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:42.067 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:42.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:42.118 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:42.118+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:42.210 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:42.209+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:42.249 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:42.249 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:42.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:42.311 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:42.311+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:42.393 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:42.393 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:42.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:42.444 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:42.444+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:42.513 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:42.513+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:42.577 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:42.577 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:42.587 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:42.619 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:42.616+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:42.715 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:42.714+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:42.762 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:42.762 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:42.771 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:42.816 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:42.815+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:42.905 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:42.905 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:42.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:42.917 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:42.916+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:43.017 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:43.017+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:43.047 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:43.048 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:43.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:43.119 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:43.118+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:43.190 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:43.190 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:43.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:43.244 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:43.243+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:43.321 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:43.320+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:43.346 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:43.346 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:43.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:43.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-06-03T05:25:43.440 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:43.439+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:43.523 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:43.522+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:43.530 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:43.531 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:43.539 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:43.624 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:43.624+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:43.723 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:43.724 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:43.724 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:43.725+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:43.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:43.825 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:43.826+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:43.877 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:43.877 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:43.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:43.926 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:43.927+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:44.023 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:44.023 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:44.028 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:44.028+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:44.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:44.150 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:44.149+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:44.170 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:44.170 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:44.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:44.229 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:44.230+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:44.331 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:44.331+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:44.358 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:44.358 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:44.358 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:44.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:44.431 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:44.431+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:44.508 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:44.508 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:44.508 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:44.508 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:44.508 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:44.515 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:44.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:44.571 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:44.571+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:44.636 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:44.636+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:44.654 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:44.654 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:44.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:44.654 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:44.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:44.654 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:44.654 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:44.654 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.784623909000004, 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:44.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:44.658 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:44.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:44.734 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:44.734+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:44.807 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:44.807 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:44.817 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:44.835 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:44.834+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:44.935 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:44.934+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:44.992 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:44.993 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:45.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:45.036 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:45.035+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:45.137 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:45.137+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:45.161 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:45.161 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:45.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:25:45.239 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:45.239+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:45.308 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:25:45.309 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:25:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:25:45.312 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:25:45.338 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:45.338+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:45.348 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:45.348 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:45.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:45.442 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:45.442+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:45.532 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:45.532 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:45.540 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:45.539+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:45.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:45.641 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:45.641+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:45.724 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:45.725 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:45.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:45.742 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:45.741+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:45.843 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:45.843+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:45.913 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:45.914 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:45.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:45.939 INFO:tasks.ceph.osd.6:Stopped 2024-06-03T05:25:45.939 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-06-03T05:25:45.939 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:25:45.939 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-06-03T05:25:45.955 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-06-03T05:25:45.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-06-03T05:25:45.966 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:45.965+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:46.045 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:46.044+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:46.134 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:46.134 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:46.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:46.145 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:46.144+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:46.246 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:46.247+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:46.255 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:25:46.287 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:46.287 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:46.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:46.383 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:46.383+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:46.449 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:46.448+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:46.473 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:46.473 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:46.483 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:46.567 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:46.567+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:46.651 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:46.650+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:46.659 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:46.659 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:46.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:46.751 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:46.752+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:46.852 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:46.853+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:46.860 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:46.860 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:46.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:46.953 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:46.954+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:47.005 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:47.006 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:47.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:47.057 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:47.056+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:47.101 INFO:teuthology.orchestra.run.smithi064.stdout:2.4 2024-06-03T05:25:47.102 INFO:teuthology.orchestra.run.smithi064.stdout:2.2 2024-06-03T05:25:47.156 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:47.156+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:47.199 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:47.199 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:47.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:47.257 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:47.257+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:47.314 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:25:47.358 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:47.358+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:47.370 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:47.370 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:47.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:47.464 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:47.464+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:47.530 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:47.556 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:47.556 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:47.559 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:47.559+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:47.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:47.568 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:25:47.568 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:25:47.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 2.2 --file /var/log/ceph/exp.2.2.6 2024-06-03T05:25:47.668 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:25:47.668 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:25:47.697 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:47.697+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:47.743 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:47.743 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:47.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:47.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:25:47.760 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:25:47.761 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:47.760+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:47.859 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:25:47.860 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:25:47.861 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:47.861+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:47.928 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:47.928 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:47.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:47.953 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:25:47.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:25:47.986 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:47.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:48.063 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:48.063+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:48.064 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:25:48.064 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:25:48.071 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:25:48.117 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:48.117 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:48.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:48.163 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:48.163+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:48.181 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:25:48.181 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:25:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:25:48.185 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:25:48.242 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:48.256 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-06-03T05:25:48.256 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-03T05:25:48.256 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:25:48.264 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:48.264+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:48.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:25:48.274 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:25:48.260+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34621) UID: 0 2024-06-03T05:25:48.274 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:25:48.262+0000 9f9e640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-06-03T05:25:48.304 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:25:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:25:48.305 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:48.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:48.364 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:48.365+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:48.480 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:48.480+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:48.499 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:48.499 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:25:48.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:25:48.499 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:25:48.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:48.499 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:48.499 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:48.499 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 89.629530381999999, 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:25:48.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:48.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:48.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:48.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:48.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:48.504 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:25:48.504 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:25:48.504 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:48.508 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:48.566 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:48.567+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:48.667 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:48.667+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:48.676 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:48.676 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:48.676 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:48.676 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:48.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:48.755 INFO:teuthology.orchestra.run.smithi064.stderr:Exporting 2.2 info 2.2( empty local-lis/les=21/22 n=0 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) 2024-06-03T05:25:48.755 INFO:teuthology.orchestra.run.smithi064.stderr:Export successful 2024-06-03T05:25:48.770 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:48.770+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:48.818 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:55.563 34626== 2024-06-03T05:25:48.818 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:55.563 34626== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:25:48.829 INFO:tasks.ceph.mon.b.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:25:48.868 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:48.869+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:48.873 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:48.873 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:48.873 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:48.873 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:48.874 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:48.883 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:48.970 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:48.970+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:49.059 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:25:49.059 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:49.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:49.071 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:49.071+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:49.172 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:49.173+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:49.250 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:49.251 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:49.251 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:49.251 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:49.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:49.273 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:49.273+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:49.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 2.2 2024-06-03T05:25:49.374 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:49.374+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:49.432 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:49.432 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:49.432 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:49.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:49.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-06-03T05:25:49.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-06-03T05:25:49.485 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:49.485+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:49.576 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:49.576+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:49.578 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:49.578 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:49.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:49.578 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:25:49.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.578 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:25:49.578 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 97.760982330999994, 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:25:49.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:25:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 97.502150528000001, 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:25:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:49.588 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:49.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:49.677 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:49.677+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:49.772 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:49.773 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:49.778 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:49.778+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:49.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:49.878 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:49.878+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:49.916 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:49.916 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:49.917 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:49.917 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:49.917 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:49.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:49.979 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:49.979+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:50.060 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:50.060 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:50.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:50.061 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:25:50.061 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:50.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:50.081 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:50.080+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:50.180 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:50.181+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:50.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:25:50.203 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:50.203 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:50.203 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:25:50.203 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:50.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:50.297 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:50.297+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:50.382 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:50.383+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:50.396 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:25:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:25:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:25:50.396 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:50.408 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:50.413 INFO:teuthology.orchestra.run.smithi064.stdout: marking collection for removal 2024-06-03T05:25:50.413 INFO:teuthology.orchestra.run.smithi064.stdout:setting '_remove' omap key 2024-06-03T05:25:50.421 INFO:teuthology.orchestra.run.smithi064.stdout:finish_remove_pgs 2.2_head removing 2.2 2024-06-03T05:25:50.424 INFO:teuthology.orchestra.run.smithi064.stdout:Remove successful 2024-06-03T05:25:50.461 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:25:50.461 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:25:50.484 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:50.484+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:50.558 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:25:50.559 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:25:50.585 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:50.585+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:50.589 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:25:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:25:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:25:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:25:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:25:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 92.924858493000002, 2024-06-03T05:25:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:25:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:25:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:25:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:25:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:25:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:25:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:25:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:25:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:25:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:25:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:25:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:25:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:25:50.596 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:25:50.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:50.654 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:25:50.654 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:25:50.686 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:50.686+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:50.747 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:25:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:25:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:25:50.748 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:25:50.748 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:25:50.749 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:25:50.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:50.787 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:50.787+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:50.845 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:25:50.845 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:25:50.888 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:50.888+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:50.934 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:50.934 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:50.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.2.6 2024-06-03T05:25:50.939 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:25:50.940 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:25:50.943 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:25:50.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:50.989 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:50.989+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:51.038 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:25:51.039 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:25:51.041 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:25:51.042 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:25:51.090 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:51.091+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:51.120 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:51.120 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:51.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:51.190 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:51.191+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:51.292 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:51.292+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:51.304 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:51.304 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:51.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:51.392 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:51.392+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:51.452 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:51.452 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:51.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:51.493 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:51.494+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:51.593 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:51.593+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:51.597 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:51.597 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:51.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:51.694 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:51.694+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:51.790 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:51.790 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:51.794 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:51.795+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:51.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:51.895 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:51.895+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:51.933 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:51.934 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:51.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:52.027 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:52.027+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:52.070 INFO:teuthology.orchestra.run.smithi064.stdout:get_pg_num_history pg_num_history pg_num_history(e35 pg_nums {1={14=1},2={21=8}} deleted_pools ) 2024-06-03T05:25:52.070 INFO:teuthology.orchestra.run.smithi064.stdout:Importing pgid 2.2 2024-06-03T05:25:52.070 INFO:teuthology.orchestra.run.smithi064.stdout:write_pg epoch 22 info 2.2( empty local-lis/les=21/22 n=0 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) 2024-06-03T05:25:52.076 INFO:teuthology.orchestra.run.smithi064.stdout:Import successful 2024-06-03T05:25:52.097 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:52.097+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:52.118 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:52.118 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:52.130 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:52.197 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:52.198+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:52.263 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:52.264 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:52.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:52.298 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:52.299+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:52.400 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:52.400+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:52.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:52.540 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:52.540+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:52.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /var/log/ceph/exp.2.2.6 2024-06-03T05:25:52.629 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:52.629+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:52.639 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:52.639 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:52.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:52.703 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:52.703+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:52.804 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:52.804+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:52.822 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:52.822 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:52.831 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:52.905 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:52.905+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:52.963 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:52.963 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:52.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:53.014 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:53.014+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:53.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:25:53.106 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:53.106+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:53.147 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:53.147 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:53.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:53.207 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:53.207+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:53.290 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:53.290 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:53.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:53.345 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:53.345+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:53.408 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:53.409+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:53.433 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:53.433 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:53.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:53.510 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:53.510+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:53.611 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:53.611+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:53.615 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:53.624 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:53.712 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:53.712+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:53.760 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:53.760 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:53.770 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:53.814 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:53.814+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:53.905 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:53.906 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:53.913 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:53.914+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:53.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:54.014 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:54.015+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:54.088 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:54.146 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:54.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:54.148 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:54.147+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:54.248 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:54.248+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:54.293 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:54.294 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:54.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:54.348 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:54.349+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:25:54.357 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-06-03T05:25:54.357 INFO:tasks.ceph.mon.b:Stopped 2024-06-03T05:25:54.357 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-06-03T05:25:54.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:54.435 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:54.435 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:54.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:54.449 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:54.449+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:54.564 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:54.564+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:54.578 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:54.579 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:54.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:25:54.672 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:54.672+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:54.752 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:54.752+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:54.761 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:54.761 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:54.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:25:54.853 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:54.853+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:54.943 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:54.943 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:54.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:25:54.953 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:54.953+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:55.036 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:25:55.091 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:55.091+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:55.126 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:55.126 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:55.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:55.180 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:55.180+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:55.256 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:55.257+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:55.269 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:55.269 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:55.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:55.363 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:55.363+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:55.452 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:55.452 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:55.458 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:55.458+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:55.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:55.559 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:55.559+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:55.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-06-03T05:25:55.637 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:55.638 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:55.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:55.692 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:55.692+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:55.761 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:55.761+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:55.787 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:55.787 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:55.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:55.861 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:55.861+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:55.962 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:55.963+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:55.969 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:55.969 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:55.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:56.064 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:56.064+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:56.151 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:56.151 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:56.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:56.164 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:56.164+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:56.265 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:56.266+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:56.299 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:56.299 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:56.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:56.367 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:56.367+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:56.441 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:56.441 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:56.450 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:56.492 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:56.492+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:56.568 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:56.568+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:56.628 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:56.628 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:56.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:56.669 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:56.669+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:56.770 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:56.770+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:56.816 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:56.816 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:56.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:56.870 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:56.870+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:56.957 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:56.958 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:56.967 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:25:57.008 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:57.007+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:57.072 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:57.072+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:57.142 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:57.142 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:57.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:25:57.193 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:57.192+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:57.275 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:57.275+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:57.326 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:57.327 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:57.336 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:25:57.376 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:57.376+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:57.477 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:57.477+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:57.480 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:57.480 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:57.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:25:57.578 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:25:57.577+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:25:57.623 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:57.623 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:57.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:25:57.657 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-06-03T05:25:57.657 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:25:57.657 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 7] 2024-06-03T05:25:57.658 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-06-03T05:25:57.658 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:25:57.678 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:57.678+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:57.685 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:57.677+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35171) UID: 0 2024-06-03T05:25:57.685 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:25:57.677+0000 b485640 -1 osd.4 38 *** Got signal Terminated *** 2024-06-03T05:25:57.781 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:57.781+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:57.783 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:57.783 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:57.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:25:57.881 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:25:57.881+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:25:57.966 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:57.966 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:57.975 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:25:57.982 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:57.981+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:58.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:25:58.094 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:58.093+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:58.150 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:58.150 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:58.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:25:58.203 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:58.203+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:58.291 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:58.287+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:58.298 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:58.298 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:58.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:25:58.385 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:58.384+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:58.482 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:58.482 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:58.486 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:58.486+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:58.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:25:58.588 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:58.587+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:58.624 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:58.625 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:58.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:25:58.689 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:58.688+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:58.774 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:58.775 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:58.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:25:58.826 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:58.825+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:58.891 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:58.892+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:58.960 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:58.960 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:58.969 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:25:58.992 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:58.992+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:59.093 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:59.092+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:59.102 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:59.102 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:59.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:25:59.196 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:59.195+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:59.285 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:59.285 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:59.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:25:59.295 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:25:59.294+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:25:59.396 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:59.396+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:59.466 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:59.466 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:59.475 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:25:59.497 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:59.497+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:59.598 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:59.599+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:59.612 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:59.613 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:59.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:25:59.699 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:25:59.698+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:25:59.771 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:59.771 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:59.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:25:59.801 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:25:59.801+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:25:59.901 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:25:59.901+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:25:59.953 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:25:59.954 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:25:59.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:00.004 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:00.003+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:00.104 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:00.103+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:00.139 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:00.139 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:00.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:00.234 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:00.233+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:00.306 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:00.306+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:00.326 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:00.326 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:00.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:00.419 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:00.418+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:00.508 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:00.508+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:00.511 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:00.511 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:00.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:00.609 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:00.609+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:00.693 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:00.693 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:00.703 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:00.744 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:00.743+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:00.810 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:00.810+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:00.849 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:00.849 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:00.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:00.912 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:00.911+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:00.993 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:00.994 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:01.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:01.047 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:01.047+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:01.114 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:01.114+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:01.136 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:01.136 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:01.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:01.215 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:01.214+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:01.316 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:01.315+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:01.318 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:01.318 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:01.328 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:01.417 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:01.417+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:01.462 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:01.462 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:01.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:01.557 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:01.556+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:01.619 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:01.619+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:01.646 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:01.646 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:01.655 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:01.720 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:01.719+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:01.804 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:01.804 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:01.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:01.821 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:01.821+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:01.922 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:01.922+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:01.985 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:01.985 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:01.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:02.038 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:02.037+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:02.124 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:02.123+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:02.127 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:02.127 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:02.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:02.225 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:02.225+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:02.311 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:02.311 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:02.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:02.365 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:02.364+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:02.442 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:02.442+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:02.472 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:02.472 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:02.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:02.566 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:02.565+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:02.632 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:02.632+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:02.657 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:02.658 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:02.667 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:02.752 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:02.751+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:02.839 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:26:02.839 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:26:02.841 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:02.840+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:02.857 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:02.857 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:02.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:02.931 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:02.932+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:02.941 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:26:02.941 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:26:03.032 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:03.031+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:03.056 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:26:03.056 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:26:03.127 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:03.128 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:03.128 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:03.128 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:03.128 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:03.133 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:03.134+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:03.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:03.166 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:26:03.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:26:03.235 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:03.234+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:03.302 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:03.303 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:03.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:03.303 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:03.303 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:03.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:03.354 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:03.353+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:03.436 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:03.436+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:03.495 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:03.495 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:03.495 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:03.495 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:03.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:03.548 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:03.547+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:03.613 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:26:03.626 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-06-03T05:26:03.626 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-03T05:26:03.626 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-06-03T05:26:03.629 INFO:tasks.ceph.mon.b:Started 2024-06-03T05:26:03.629 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-06-03T05:26:03.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:26:03.638 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:03.638+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:26:03.660 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:03.661 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:03.673 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:03.758 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:03.757+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:03.788 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:26:03.904 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:26:03.905 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 106.250320152, 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:03.920 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:03.923 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:03.921+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:03.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:03.941 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:03.941+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:04.045 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:04.045+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:04.061 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:04.061 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:04.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:04.143 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:04.142+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:04.146 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:26:04.240 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:26:04.240 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:26:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:26:04.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:26:04.244 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:04.243+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:04.277 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:04.277 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:04.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:04.344 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:04.343+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:04.383 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:26:04.446 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:04.445+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:04.473 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.603940839, 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:04.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 124.428984507, 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:04.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:04.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 124.42802207299999, 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.485 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:04.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:04.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:04.490 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 124.42723207, 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:04.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 124.39257820900001, 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:04.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 124.391888712, 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:04.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.506 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 124.391215363, 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:04.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.512 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 124.31878224800001, 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:04.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:04.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.560795056, 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.312144735, 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.523 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:04.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.528 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.31089934800001, 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:04.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.533 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.30983983900001, 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.534 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:04.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:04.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.276991444, 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:04.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:04.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:04.544 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.27590631299999, 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.27488929499999, 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.550 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:04.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:04.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.255105778, 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:04.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 119.130497785, 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:04.560 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:04.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 119.129678121, 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.566 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:04.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 119.12897717200001, 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:04.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:04.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:04.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 112.72192344, 2024-06-03T05:26:04.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:04.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:04.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:04.580 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:04.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:04.582 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:04.581+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:04.683 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:04.682+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:04.717 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:04.717 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:04.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:04.783 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:04.783+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:04.862 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:04.862 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:04.862 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:04.862 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:04.862 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:04.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:04.885 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:04.884+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:04.985 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:04.984+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:05.010 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 106.140458415, 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:05.015 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:05.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:05.086 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:05.085+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:05.154 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:05.154 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:05.155 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:05.155 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:05.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:05.207 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:05.204+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:05.288 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:05.288+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:05.337 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:05.337 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:05.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:05.392 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:05.391+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:05.483 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:05.483 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:05.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:05.484 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:05.484 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:05.490 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:05.490+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:05.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:05.590 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:05.590+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:05.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:26:05.669 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:05.669 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:05.669 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:05.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:05.721 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:05.719+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:05.793 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:05.793+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:05.886 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:05.887 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:05.895 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:05.894+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:05.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:05.995 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:05.994+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:06.078 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:06.078 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:06.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:06.078 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:06.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.078 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:06.078 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:06.078 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 114.260634897, 2024-06-03T05:26:06.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:06.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:06.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:06.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.083 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:06.083 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:06.083 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 114.001803094, 2024-06-03T05:26:06.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:06.083 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:06.083 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:06.083 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:06.084 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:06.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:06.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:06.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:06.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:06.088 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:06.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:06.131 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:06.130+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:06.196 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:06.196+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:06.245 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:26:06.266 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:06.266 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:06.266 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:06.266 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:06.276 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:06.297 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:06.297+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:06.398 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:06.397+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:06.464 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:06.464 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:06.464 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:06.464 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:06.464 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:06.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:06.514 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:06.513+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:06.600 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:06.599+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:06.650 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:06.650 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:06.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:06.650 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:06.650 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:06.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:06.700 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:06.700+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:06.713 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:26:06.713 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:26:06.801 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:06.801+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:06.839 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:26:06.840 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:26:06.858 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:06.859 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:06.859 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:06.859 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:06.868 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:06.910 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:06.908+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:06.945 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:26:06.945 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:26:07.020 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:26:07.057 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:26:07.057 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:26:07.058 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:07.057+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:07.060 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:26:07.098 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:07.098 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:07.105 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:07.105+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:07.108 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:07.152 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:26:07.152 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:26:07.155 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:26:07.206 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:07.206+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:07.248 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:26:07.248 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:26:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:26:07.252 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:26:07.282 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:07.283 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:07.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:07.341 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:07.341+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:07.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:26:07.428 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:07.428 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:07.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:07.442 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:07.441+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:07.508 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:07.507+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:07.609 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:07.609+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:07.610 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:07.610 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:07.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:07.710 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:07.710+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:07.796 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:07.796 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:07.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-06-03T05:26:07.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:07.811 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:07.811+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:07.860 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:20.848 42879== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:07.896 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:20.884 42879== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:07.913 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:07.912+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:07.925 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:20.914 42879== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:07.942 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:07.943 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:07.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:07.954 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:48.398 46030== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:07.959 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:48.404 46030== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:07.981 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:48.427 46030== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:08.013 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:08.013+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:08.088 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:08.088 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:08.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:08.139 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:08.139+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:08.155 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:26:08.214 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:08.213+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:08.278 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:08.278 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:08.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:08.315 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:08.315+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:08.416 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:08.416+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:08.421 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:08.421 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:08.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:08.517 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:08.517+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:08.604 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:08.604 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:08.614 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:08.618 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:08.619+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:08.719 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:08.719+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:08.786 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:08.786 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:08.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:08.840 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:08.839+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:08.922 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:08.921+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:08.928 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:08.928 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:08.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:09.024 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:09.023+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:09.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:26:09.124 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:09.123+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:09.135 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:09.135 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:09.147 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:09.232 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:09.231+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:09.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:26:09.326 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:09.326+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:09.332 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:09.332 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:09.341 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:09.426 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:09.426+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:09.501 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:09.501 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:09.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:09.527 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:09.527+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:09.544 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:26:09.544 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:26:09.628 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:09.628+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:09.640 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:26:09.640 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:26:09.673 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:09.673 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:09.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:09.769 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:09.768+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:09.770 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:26:09.771 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:26:09.829 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:09.828+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:09.857 INFO:tasks.ceph.osd.4:Stopped 2024-06-03T05:26:09.857 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-06-03T05:26:09.857 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:26:09.857 INFO:tasks.thrashosds.thrasher:remote for exp osd 6 is ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:26:09.857 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-06-03T05:26:09.872 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-06-03T05:26:09.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-06-03T05:26:09.891 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:09.891 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:09.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:09.891 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:09.891 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:09.896 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:26:09.896 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:26:09.901 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:09.901 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:26:09.943 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:09.942+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:10.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:26:10.009 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:26:10.011 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:26:10.031 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:10.031+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:10.083 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:10.083 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:10.083 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:10.083 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:10.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:10.106 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:26:10.107 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:26:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:26:10.112 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:26:10.133 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:10.132+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:10.234 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:10.234+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:26:10.251 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:10.251 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:10.252 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:10.252 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:10.252 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:10.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:10.335 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:10.335+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:10.436 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:10.435+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 112.77638508, 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:10.441 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:10.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:10.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:10.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:10.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:10.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:10.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:10.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:10.445 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:10.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:10.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:10.445 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:10.445 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:10.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:10.536 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:10.536+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:10.606 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:10.607 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:10.607 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:10.607 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:10.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:10.639 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:10.638+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:10.739 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:10.739+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:10.795 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:10.796 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:10.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:10.839 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:10.839+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:10.988 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:10.988 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 132.119291371, 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:10.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:10.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:10.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:10.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:10.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.94433503900001, 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:10.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:10.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:10.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.94337260500001, 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.94258260199999, 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.90792874100001, 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.90723924400001, 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.906565895, 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.83413278, 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 130.076145588, 2024-06-03T05:26:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.82749526699999, 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.82624988000001, 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.82519037099999, 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.79234197599999, 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.79125684499999, 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.79023982699999, 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.77045630999999, 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.645848317, 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.645028653, 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.64432770400001, 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 119.237273972, 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.092 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.092 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:11.093 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:10.989+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:11.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:11.094 INFO:teuthology.orchestra.run.smithi064.stdout:2.4 2024-06-03T05:26:11.094 INFO:teuthology.orchestra.run.smithi064.stdout:2.2 2024-06-03T05:26:11.095 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:11.093+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:11.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:26:11.194 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:11.194+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:11.230 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:11.230 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:11.230 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:11.230 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:11.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:11.323 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:11.323+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:11.369 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:07.675 46834== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:11.395 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:11.396+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:11.414 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:11.414 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:11.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:11.508 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:11.508+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:11.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 2.2 --file /var/log/ceph/exp.2.2.6 2024-06-03T05:26:11.596 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:11.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 112.73641557400001, 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:11.610 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:11.610 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:07.916 46834== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:11.616 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:11.698 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:11.697+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:11.758 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:11.758 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:11.758 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:11.758 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:11.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:11.801 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:11.799+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:11.900 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:11.900+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:11.951 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:26:11.956 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:11.956 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:11.956 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:11.956 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:11.956 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:11.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:12.009 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:12.009+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:12.102 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:12.102+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:12.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:26:12.144 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:12.145 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:12.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:12.203 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:12.203+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:12.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:26:12.304 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:12.304+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:12.364 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:12.365 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:12.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:12.402 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:26:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:26:12.406 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:12.405+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:12.504 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:26:12.504 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:26:12.505 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:12.504+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:12.542 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:12.542 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:12.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:12.620 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:26:12.620 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:26:12.621 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:12.621+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:Exporting 2.2 info 2.2( empty local-lis/les=21/22 n=0 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) 2024-06-03T05:26:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:Export successful 2024-06-03T05:26:12.707 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:12.706+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:12.720 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:26:12.720 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:26:12.723 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:26:12.725 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:12.726 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:12.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:12.816 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:26:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:26:12.816 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:12.816+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:12.818 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:26:12.909 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:12.908+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:12.911 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:12.911 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:12.912 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:26:12.913 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:26:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:26:12.917 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:26:12.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:13.010 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:13.009+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:13.095 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:13.095 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:13.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:13.147 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:13.146+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:13.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 2.2 2024-06-03T05:26:13.212 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:13.212+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:13.279 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:13.279 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:13.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:13.332 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:13.330+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:13.414 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:13.414+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:13.469 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:13.469 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:13.479 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:13.504 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:26:13.529 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:13.528+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:13.615 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:13.614+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:13.617 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:13.617 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:13.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:13.716 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:13.715+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:13.804 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:13.804 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:13.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:13.857 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:13.857+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:13.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-06-03T05:26:13.948 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:13.948+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:13.963 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:13.963 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:13.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:14.057 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:14.057+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:14.139 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:14.139+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:14.150 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:14.151 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:14.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:14.219 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:14.219+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:14.321 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:14.320+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:14.336 INFO:teuthology.orchestra.run.smithi064.stdout: marking collection for removal 2024-06-03T05:26:14.336 INFO:teuthology.orchestra.run.smithi064.stdout:setting '_remove' omap key 2024-06-03T05:26:14.342 INFO:teuthology.orchestra.run.smithi064.stdout:finish_remove_pgs 2.2_head removing 2.2 2024-06-03T05:26:14.342 INFO:teuthology.orchestra.run.smithi064.stdout:Remove successful 2024-06-03T05:26:14.353 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:14.353 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:14.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:14.421 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:14.421+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:14.503 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:14.503 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:14.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:14.522 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:14.521+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:14.623 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:14.623+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:14.648 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:14.648 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:14.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:14.724 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:14.724+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:14.797 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:14.797 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:14.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:14.825 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:14.825+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:14.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-06-03T05:26:14.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:26:14.940 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:14.941 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:14.941 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:14.941+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:14.950 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:14.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:26:15.007 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:11.313 46834== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:15.036 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:15.036+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:15.067 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:11.373 46834== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:15.128 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:15.128+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:15.159 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:15.159 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:15.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:15.223 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:26:15.223 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:26:15.229 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:15.229+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:15.310 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:11.616 46834== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:15.319 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:26:15.319 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:26:15.329 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:15.330+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:15.361 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:15.361 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:15.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:15.361 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:15.361 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:15.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:15.413 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:26:15.413 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:26:15.430 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:15.430+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:15.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:26:15.537 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:26:15.538 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:26:15.538 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:15.538+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:15.541 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:26:15.577 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:15.577 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:15.577 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:15.578 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:15.587 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:15.658 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:26:15.658 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:26:15.658 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:15.659+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:15.732 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:15.732+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:15.780 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:15.780 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:15.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:15.832 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:15.833+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:15.933 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:15.933+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:15.972 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:15.972 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:15.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:15.972 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:15.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.972 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:15.972 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:15.972 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 124.154865313, 2024-06-03T05:26:15.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:15.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:15.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:15.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 123.89603351, 2024-06-03T05:26:15.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:15.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:15.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:15.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:15.980 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:15.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:16.002 INFO:teuthology.orchestra.run.smithi064.stdout:2.6 2024-06-03T05:26:16.002 INFO:teuthology.orchestra.run.smithi064.stdout:2.5 2024-06-03T05:26:16.002 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:26:16.017 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-06-03T05:26:16.017 INFO:tasks.mon_recovery:stopping all monitors 2024-06-03T05:26:16.017 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-03T05:26:16.017 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:26:16.059 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:16.059+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:16.074 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:26:16.059+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 46832) UID: 0 2024-06-03T05:26:16.074 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:26:16.061+0000 9f9e640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-06-03T05:26:16.135 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:16.135+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:16.140 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:16.140 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:16.140 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:16.140 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:16.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:16.174 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:26:16.278 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:16.278+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:16.278 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:26:16.279 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:26:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:26:16.282 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:26:16.331 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:16.332 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:16.337 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:16.338+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:16.341 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:16.438 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:16.439+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:16.514 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:16.514 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:16.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:16.515 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:16.515 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:16.519 INFO:tasks.thrashosds.thrasher:Moving pg 2.2 from osd.6 to osd.4 2024-06-03T05:26:16.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.2.6 2024-06-03T05:26:16.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:16.568 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:16.568+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:16.639 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:16.640+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:16.662 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:16.663 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:16.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:16.757 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:16.757+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:16.820 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:13.125 46834== 2024-06-03T05:26:16.820 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:13.125 46834== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:26:16.829 INFO:tasks.ceph.mon.b.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:26:16.842 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:16.842+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:16.849 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:16.850 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:16.850 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:16.850 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:16.850 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:16.859 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:16.943 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:16.943+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 119.375227845, 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:17.044 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:17.046 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:17.046+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:17.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:17.147 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:17.147+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:17.184 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:17.184 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:17.184 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:17.184 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:17.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:17.248 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:17.248+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:17.349 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:17.349+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:17.366 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:17.366 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:17.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:17.450 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:17.450+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:17.551 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:17.551+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 138.68950468099999, 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:17.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:17.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:17.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:17.565 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 137.51454834899999, 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:17.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 137.51358591499999, 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.571 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:17.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 137.512795912, 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:17.576 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.581 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 137.47814205099999, 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:17.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:17.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 137.477452554, 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:17.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 137.47677920500001, 2024-06-03T05:26:17.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 137.40434608999999, 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.598 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.603 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.64635889799999, 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:17.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:17.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.397708577, 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:17.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:17.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.39646318999999, 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:17.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:17.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.395403681, 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:17.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.362555286, 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.625 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:17.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:17.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.36147015500001, 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.36045313700001, 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.636 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:17.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 136.34066962, 2024-06-03T05:26:17.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:17.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:17.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 132.21606162699999, 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.647 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 132.21524196300001, 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.652 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:17.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 132.21454101399999, 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:17.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.807487282, 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:17.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:17.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:17.666 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:17.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:17.668 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:17.668+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:17.682 INFO:teuthology.orchestra.run.smithi064.stdout:get_pg_num_history pg_num_history pg_num_history(e38 pg_nums {1={14=1},2={21=8}} deleted_pools ) 2024-06-03T05:26:17.683 INFO:teuthology.orchestra.run.smithi064.stdout:Importing pgid 2.2 2024-06-03T05:26:17.689 INFO:teuthology.orchestra.run.smithi064.stdout:write_pg epoch 22 info 2.2( empty local-lis/les=21/22 n=0 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) 2024-06-03T05:26:17.691 INFO:teuthology.orchestra.run.smithi064.stdout:Import successful 2024-06-03T05:26:17.770 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:17.770+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:17.816 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:17.816 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:17.816 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:17.816 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:17.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:17.870 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:17.870+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:17.971 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:17.971+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:18.000 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:18.000 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:18.000 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:18.000 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:18.000 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:18.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:18.072 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:18.072+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:18.146 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:18.146 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:18.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:18.146 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:18.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:18.146 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:18.146 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 119.275223674, 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:18.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:18.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:18.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:18.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:18.151 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:18.151 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:18.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:18.172 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:18.173+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:18.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /var/log/ceph/exp.2.2.6 2024-06-03T05:26:18.278 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:18.278+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:18.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:26:18.300 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:18.300 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:18.300 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:18.301 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:18.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:18.396 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:18.396+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:18.475 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:18.476+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:18.507 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:18.507 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:18.507 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:18.507 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:18.507 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:18.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:18.602 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:18.602+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:18.678 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:18.678+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:18.691 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:18.691 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:18.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:18.784 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:18.784+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:18.873 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:18.874 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:18.874 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:18.874 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:18.879 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:18.879+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:18.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:18.979 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:18.980+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:19.055 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:19.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:19.055 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:19.055 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:19.055 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:19.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:19.106 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:19.106+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:19.181 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:19.182+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:19.242 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:19.242 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:19.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:19.242 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:19.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.242 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:19.242 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:19.242 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 127.424634248, 2024-06-03T05:26:19.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:19.243 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:19.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:19.243 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:19.243 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:19.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:19.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:19.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:19.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:19.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:19.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:19.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:19.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:19.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:19.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:19.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:19.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:19.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:19.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:19.247 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 127.165802445, 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:19.248 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:19.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:19.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:19.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:19.253 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:19.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:19.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:19.253 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:19.253 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:19.254 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:19.297 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:19.296+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:19.383 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:19.383+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:19.429 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:19.429 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:19.430 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:19.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:19.484 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:19.484+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:19.575 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:19.575 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:19.575 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:19.575 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:19.575 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:19.584 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:19.584+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:19.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:19.684 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:19.685+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:19.760 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:19.760 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:19.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:19.760 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:19.761 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:19.769 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:19.787 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:19.787+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:19.888 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:19.888+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:19.944 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:19.944 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:19.945 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:19.945 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:19.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-06-03T05:26:19.954 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:19.995 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:19.995+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:20.091 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:20.092+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:20.133 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:20.133 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:20.133 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:20.133 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:20.133 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:20.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:20.227 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:20.227+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:20.302 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:20.302+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:20.320 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:20.320 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 122.65417669999999, 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:20.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.50620296, 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:20.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:20.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:20.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:20.328 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:20.329 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:20.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:20.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:20.329 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:20.329 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:20.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:20.394 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:20.395+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:20.476 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:20.477 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:20.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:20.495 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:20.496+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:20.597 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:20.597+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:20.619 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:20.619 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:20.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:20.698 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:20.698+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:20.811 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 141.942190044, 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.76723371200001, 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.766271278, 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.76548127500001, 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.730827414, 2024-06-03T05:26:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.73013791700001, 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.729464568, 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:20.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:20.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.852 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.657031453, 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.853 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:20.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:20.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.899044261, 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:20.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:20.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:20.863 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.65039393999999, 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:20.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.649148553, 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:20.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:20.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.64808904399999, 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:20.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:20.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.61524064899999, 2024-06-03T05:26:20.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:20.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.61415551799999, 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:20.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:20.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.61313849999999, 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:20.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:20.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.59335498300001, 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.895 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:20.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.46874699, 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.900 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:20.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.46792732599999, 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:20.905 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:20.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.910 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.467226377, 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:20.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.915 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.06017264499999, 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:20.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:20.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:20.919 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:20.920 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:20.810+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:20.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:20.922 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:20.923+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:21.023 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:21.023+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:21.094 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:21.095 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:21.095 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:21.095 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:21.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:21.146 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:21.145+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:21.225 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:21.225+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:21.277 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:21.277 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:21.277 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:21.277 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:21.277 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:21.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:21.330 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:21.329+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:21.427 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:21.427+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:21.464 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.59327043099999, 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.4753112850000001, 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:21.473 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:21.474 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:21.528 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:21.528+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:21.621 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:21.622 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:21.622 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:21.622 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:21.629 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:21.629+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:21.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:21.730 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:21.730+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:21.804 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:21.804 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:21.814 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:21.831 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:21.831+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:21.931 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:21.932+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:21.989 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:21.989 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:21.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:22.032 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:22.033+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:26:22.118 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-06-03T05:26:22.119 INFO:tasks.ceph.mon.b:Stopped 2024-06-03T05:26:22.119 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-03T05:26:22.119 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:26:22.132 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:22.133+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:22.136 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:22.136 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:22.136 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:22.136 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:22.139 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:26:22.125+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 46025) UID: 0 2024-06-03T05:26:22.139 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:26:22.126+0000 9f9e640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-06-03T05:26:22.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:22.234 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:22.234+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:22.335 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:22.335+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:22.345 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:22.345 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:22.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:22.442 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:22.442+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:22.536 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:22.536 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 130.71910112, 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:22.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:22.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:22.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:22.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 130.46026931700001, 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:22.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:22.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:22.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:22.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:22.545 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:22.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:22.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:22.545 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:22.545 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:22.545 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:22.537+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:22.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:22.637 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:22.637+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:22.672 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:03.116 46030== 2024-06-03T05:26:22.672 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:03.116 46030== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:26:22.682 INFO:tasks.ceph.mon.c.smithi038.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:26:22.724 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:22.724 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:22.725 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:22.725 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:22.734 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:22.738 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:22.738+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:22.840 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:22.840+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:22.908 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:22.908 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:22.908 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:22.908 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:22.908 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:22.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:22.941 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:22.941+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:23.042 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:23.042+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:23.053 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:23.054 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:23.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:23.143 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:23.143+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:23.195 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:23.195 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:23.195 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:23.195 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:23.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:23.246 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:23.246+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:23.279 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [6, 4] live_osds: [0, 1, 2, 3, 5, 7] 2024-06-03T05:26:23.279 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:26:23.279 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-03T05:26:23.279 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-06-03T05:26:23.344 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:23.344+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:23.379 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:23.379 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:23.379 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:23.379 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:23.379 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:23.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:23.445 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:23.445+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:23.546 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:23.546+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:23.547 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 125.88072136, 2024-06-03T05:26:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7327476199999996, 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:23.556 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:23.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:23.647 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:23.647+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:23.705 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:23.705 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:23.705 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:23.705 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:23.714 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:23.747 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:23.747+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:23.848 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:23.848+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:23.888 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:23.888 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:23.888 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:23.888 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:23.889 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:23.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:23.948 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:23.949+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:24.058 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:24.059+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.25509875, 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:24.124 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:24.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.129 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.08014241800001, 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:24.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.134 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.07917998400001, 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:24.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:24.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.07838998099999, 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:24.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:24.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.04373612000001, 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.04304662300001, 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:24.151 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:24.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 144.042373274, 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:24.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:24.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 143.969940159, 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.162 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:24.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 143.211952967, 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.96330264599999, 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:24.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.96205725900001, 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:24.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.96099774999999, 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:24.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.188 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.92814935499999, 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:24.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.92706422399999, 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:24.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.198 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.92604720599999, 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:24.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:24.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.90626368900001, 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.204 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:24.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:24.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 138.781655696, 2024-06-03T05:26:24.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:24.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.214 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 138.780836032, 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:24.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 138.780135083, 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:24.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:24.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 132.373081351, 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:24.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.228 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:24.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:24.230 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:24.230+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:24.331 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:24.331+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:24.406 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:24.407 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:24.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:24.432 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:24.432+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:24.536 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:24.536+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:24.551 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:24.551 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:24.551 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:24.551 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:24.551 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:24.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:24.646 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:24.646+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:24.736 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:24.736+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:24.739 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:24.739 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:24.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:24.739 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:24.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.739 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:24.739 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:24.739 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.867877726, 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.743 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.7499185800000001, 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:24.747 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:24.748 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:24.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:24.837 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:24.838+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:24.894 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:24.894 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:24.894 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:24.894 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:24.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:24.946 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:24.945+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:25.040 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:25.040+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:25.077 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:25.077 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:25.077 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:25.077 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:25.077 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:25.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:25.171 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:25.171+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:25.242 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:25.242+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:25.261 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:25.261 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:25.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:25.261 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:25.261 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:25.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:25.356 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:25.356+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:25.444 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:25.444+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:25.447 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:25.447 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:25.447 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:25.447 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:25.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:25.545 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:25.545+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:25.630 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:25.630 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:25.631 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:25.631 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:25.631 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:25.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:25.682 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:25.682+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:25.746 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:25.747+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:25.817 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 133.99997403699999, 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:25.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:25.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 133.74114223399999, 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:25.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:25.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:25.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:25.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:25.827 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:25.827 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:25.827 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:25.827 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:25.827 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:25.827 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:25.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:25.847 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:25.847+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:25.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-06-03T05:26:25.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-06-03T05:26:25.947 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:25.948+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:26.002 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:26.003 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:26.003 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:26.003 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:26.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:26.054 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:26.053+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:26.150 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:26.150+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:26.184 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:26.184 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:26.184 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:26.184 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:26.184 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:26.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:26.279 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:26.279+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:26.351 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:26.352+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:26.368 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:26.369 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:26.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:26.465 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:26.465+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:26.554 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:26.554+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:26.555 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:26.555 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:26.555 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:26.555 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:26.565 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:26.655 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:26.655+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:26.738 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:26.739 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:26.747 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:26.789 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:26.789+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:26.857 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:26.857+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:26.925 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 129.25917358000001, 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.1111998399999994, 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:26.934 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:26.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:26.977 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:26.977+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:27.059 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:27.059+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:27.081 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:27.081 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:27.081 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:27.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:27.160 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:27.160+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:27.225 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:27.225 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:27.225 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:27.225 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:27.225 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:27.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:27.261 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:27.262+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:27.363 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:27.363+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 148.548876111, 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:27.418 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 147.373919779, 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 147.372957345, 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 147.37216734200001, 2024-06-03T05:26:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 147.337513481, 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 147.33682398400001, 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 147.336150635, 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 147.26371752, 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 146.505730328, 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.460 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 146.25708000700001, 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 146.25583462, 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 146.25477511099999, 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 146.22192671600001, 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 146.22084158499999, 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 146.21982456699999, 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 146.20004105000001, 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.075433057, 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.07461339299999, 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.073912444, 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.66685871199999, 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:27.521 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:27.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:27.522 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:27.522+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:27.642 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:27.642+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:27.656 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:27.656 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:27.657 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:27.657 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:27.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:27.751 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:27.751+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:27.825 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:27.826+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:27.840 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:27.840 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:27.840 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:27.841 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:27.841 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:27.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:27.934 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:27.934+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:28.027 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:28.027+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:28.028 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:28.028 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.15594850700001, 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:28.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:28.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:28.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:28.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0379893609999993, 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:28.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:28.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:28.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:28.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:28.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:28.038 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:28.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:28.127 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:28.128+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:28.184 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:28.184 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:28.184 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:28.184 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:28.193 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:26:28.221 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-06-03T05:26:28.221 INFO:tasks.ceph.mon.c:Stopped 2024-06-03T05:26:28.222 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-03T05:26:28.222 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:26:28.235 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:28.235+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:28.251 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-06-03T05:26:28.238+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 42873) UID: 0 2024-06-03T05:26:28.251 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-06-03T05:26:28.240+0000 9f9e640 -1 mon.a@0(probing) e1 *** Got Signal Terminated *** 2024-06-03T05:26:28.329 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:28.329+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:28.391 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:28.391 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:28.391 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:28.391 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:28.391 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:28.400 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:28.430 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:28.430+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:28.531 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:28.531+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:28.588 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:28.588 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:28.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:28.588 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:28.588 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:28.597 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:28.641 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:28.641+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:28.733 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:28.733+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:28.733 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:28.733 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:28.733 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:28.733 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:28.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:28.833 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:28.833+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:28.876 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:41.864 42879== 2024-06-03T05:26:28.876 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:41.864 42879== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:26:28.888 INFO:tasks.ceph.mon.a.smithi038.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:26:28.923 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:28.923 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:28.923 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:28.923 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:28.923 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:28.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:28.934 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:28.935+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:29.034 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:29.034+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:29.113 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:29.113 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:29.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:29.113 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:29.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.113 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 137.29587589600001, 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:29.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:29.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:29.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 137.03704409299999, 2024-06-03T05:26:29.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:29.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:29.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:29.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:29.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:29.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:29.123 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:29.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:29.171 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:29.169+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:29.237 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:29.236+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:29.261 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:29.261 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:29.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:29.355 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:29.355+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:29.439 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:29.439+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:29.450 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:29.450 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:29.450 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:29.450 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:29.450 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:29.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:29.540 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:29.539+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:29.634 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:29.634 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:29.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:29.634 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:29.635 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:29.640 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:29.640+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:29.643 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:29.741 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:29.741+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:29.818 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:29.818 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:29.818 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:29.818 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:29.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:29.869 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:29.869+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:29.942 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:29.943+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:30.002 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:30.002 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:30.002 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:30.002 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:30.003 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:30.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:30.044 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:30.043+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:30.145 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:30.145+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:30.189 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:30.189 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:30.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 132.52283283899999, 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:30.190 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:30.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.374859099, 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:30.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:30.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:30.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:30.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:30.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:30.198 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:30.198 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:30.198 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:30.198 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:30.198 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:30.198 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:30.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:30.246 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:30.245+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:30.346 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:30.346 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:30.346 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:30.346 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:30.347 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:30.346+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:30.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:30.448 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:30.448+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:30.530 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:30.530 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:30.530 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:30.530 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:30.530 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:30.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:30.548 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:30.549+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:30.650 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:30.650+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:30.724 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 151.85489012799999, 2024-06-03T05:26:30.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:30.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:30.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:30.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.679933796, 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:30.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:30.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.678971362, 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.735 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:30.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:30.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:30.740 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.67818135900001, 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:30.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.745 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.643527498, 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.746 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:30.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:30.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.642838001, 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:30.751 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:30.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.64216465199999, 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:30.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:30.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:30.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 150.569731537, 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.762 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:30.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.767 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.81174434499999, 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:30.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.56309402400001, 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:30.773 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:30.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.561848637, 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:30.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:30.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.56078912800001, 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.789 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.52794073300001, 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:30.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.52685560200001, 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:30.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:30.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:30.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.52583858400001, 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.800 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:30.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.50605506700001, 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.806 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:30.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.811 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.38144707399999, 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:30.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:30.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:30.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.38062740999999, 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:30.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.379926461, 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.822 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:30.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 138.97287272899999, 2024-06-03T05:26:30.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:30.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:30.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:30.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:30.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.831 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:30.831 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:30.831 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:30.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:30.833 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:30.833+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:30.952 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:30.951+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:30.967 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:30.967 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:30.967 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:30.967 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:30.977 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:31.035 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:31.035+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:31.137 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:31.137+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:31.150 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:31.150 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:31.150 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:31.150 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:31.151 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:31.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:31.238 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:31.238+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:31.299 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:31.299 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 132.42836333299999, 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:31.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:31.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:31.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.310404187, 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:31.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:31.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:31.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:31.308 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:31.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:31.339 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:31.338+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:31.440 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:31.440+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:31.457 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:31.457 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:31.457 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:31.457 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:31.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:31.541 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:31.541+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:31.603 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:31.603 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:31.603 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:31.604 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:31.604 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:31.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:31.655 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:31.655+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:31.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-06-03T05:26:31.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-06-03T05:26:31.742 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:31.743+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:31.788 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:31.788 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:31.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:31.788 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:31.789 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:31.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:31.882 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:31.882+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:31.944 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:31.944+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:31.970 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:31.971 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:31.971 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:31.971 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:31.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:32.045 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:32.044+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:32.145 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:32.146+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:32.155 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:32.156 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:32.165 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:32.247 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:32.246+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 140.52617026999999, 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:32.344 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:32.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 140.267338467, 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:32.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:32.353 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:32.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:32.355 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:32.355+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:32.456 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:32.456+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:32.532 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:32.532 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:32.532 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:32.532 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:32.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:32.584 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:32.583+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:32.658 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:32.658+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:32.715 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:32.715 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:32.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:32.767 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:32.766+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:32.860 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:32.860+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:32.898 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:32.899 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:32.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:32.899 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:32.899 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:32.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:32.992 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:32.992+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:33.062 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:33.061+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:33.085 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:33.086 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:33.086 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:33.086 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:33.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:33.178 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:33.178+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:33.264 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:33.264+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:33.267 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:33.267 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:33.267 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:33.267 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:33.267 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:33.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:33.364 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:33.365+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:33.455 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 135.78936254999999, 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:33.456 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:33.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:33.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:33.460 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.64138881, 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:33.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:33.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:33.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:33.464 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:33.465 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:33.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:33.466 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:33.466+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:33.568 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:33.568+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:33.612 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:33.612 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:33.612 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:33.612 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:33.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:33.669 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:33.669+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:33.766 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:33.766 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:33.766 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:33.766 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:33.766 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:33.770 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:33.771+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:33.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:33.871 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:33.872+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:33.961 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:33.961 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 155.09201328899999, 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:33.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:33.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.917056957, 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:33.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:33.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:33.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.916094523, 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:33.973 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:33.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.978 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.91530452000001, 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:33.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:33.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:33.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.880650659, 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:33.984 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:33.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:33.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.989 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.879961162, 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.990 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:33.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:33.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:33.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:33.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.87928781299999, 2024-06-03T05:26:33.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:33.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:33.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:33.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:34.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.806854698, 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.001 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:34.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.006 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 153.04886750599999, 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:34.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:34.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:34.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.80021718500001, 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.012 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:34.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:34.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:34.017 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.798971798, 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.79791228900001, 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.76506389400001, 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.76397876300001, 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.76296174500001, 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.743178228, 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 148.61857023499999, 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 148.61775057099999, 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 148.617049622, 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.20999588999999, 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.071 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:34.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:34.073 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:34.073+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:34.174 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:34.175+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:34.207 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:34.207 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:34.207 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:34.207 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:34.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:34.276 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:34.276+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:26:34.324 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-06-03T05:26:34.324 INFO:tasks.ceph.mon.a:Stopped 2024-06-03T05:26:34.324 INFO:tasks.mon_recovery:forming a minimal quorum for ['b', 'c', 'a'], then adding monitors 2024-06-03T05:26:34.325 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-03T05:26:34.325 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-06-03T05:26:34.327 INFO:tasks.ceph.mon.b:Started 2024-06-03T05:26:34.328 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-03T05:26:34.328 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-06-03T05:26:34.331 INFO:tasks.ceph.mon.c:Started 2024-06-03T05:26:34.331 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-06-03T05:26:34.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:26:34.350 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:34.351 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:34.351 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:34.351 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:34.351 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:34.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:34.376 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:34.376+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:34.477 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:34.477+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:34.542 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:34.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.670933461, 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:34.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:34.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.552974315, 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:34.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:34.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:34.551 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:34.552 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:34.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:34.578 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:34.577+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:34.679 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:34.679+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:34.708 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:34.708 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:34.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:34.802 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:34.802+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:34.893 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:34.893+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:34.908 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:34.908 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:34.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:34.981 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:34.980+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:35.082 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:35.083+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:35.119 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:35.119 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:35.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:35.212 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:35.212+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:35.285 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:35.285+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:35.324 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:35.324 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:35.324 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:35.324 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:35.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:35.418 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:35.418+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:35.487 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:35.487+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:35.526 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:35.526 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:35.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:35.621 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:35.621+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:35.696 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:35.696+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:35.716 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:35.716 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:35.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:35.716 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:35.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.716 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:35.716 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:35.716 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 143.89820965600001, 2024-06-03T05:26:35.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:35.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:35.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:35.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:35.720 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 143.63937785300001, 2024-06-03T05:26:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:35.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:35.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:35.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:35.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:35.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:35.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:35.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:35.726 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:35.726 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:35.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:35.790 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:35.789+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:35.891 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:35.891+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:35.908 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:35.908 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:35.908 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:35.908 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:35.918 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:36.003 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:36.003+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:36.096 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:36.096+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:36.111 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:36.112 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:36.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:36.206 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:36.206+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:36.294 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:36.295+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:36.314 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:36.315 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:36.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:36.315 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:36.315 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:36.324 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:36.396 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:36.395+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:36.497 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:36.496+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:36.501 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:36.501 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:36.501 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:36.501 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:36.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:36.598 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:36.599+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:36.648 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:36.648 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:36.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:36.699 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:36.699+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:36.817 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:36.817+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:36.837 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:36.837 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:36.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:36.837 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:36.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.837 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 139.17070199200001, 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:36.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:36.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:36.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:36.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.022728252, 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:36.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:36.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:36.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:36.846 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:36.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:36.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:36.846 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:36.846 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:36.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:36.902 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:36.902+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:36.998 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:36.998 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:36.998 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:37.003 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:37.003+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:37.008 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:37.104 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:37.103+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:37.141 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:37.142 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:37.142 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:37.142 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:37.142 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:37.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:37.204 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:37.205+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:37.311 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:37.310+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.50733163000001, 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 157.33237529799999, 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 157.33141286399999, 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 157.33062286099999, 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 157.29596900000001, 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 157.29527950299999, 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 157.29460615400001, 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 157.22217303900001, 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.46418584700001, 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.215535526, 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.21429013900001, 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:37.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:37.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:37.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.21323063, 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:37.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:37.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:37.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.180382235, 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:37.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.449 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.179297104, 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:37.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:37.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.178280086, 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:37.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:37.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 156.15849656899999, 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:37.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:37.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.03388857600001, 2024-06-03T05:26:37.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:37.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:37.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.033068912, 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:37.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.03236796300001, 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:37.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.480 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.625314231, 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:37.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:37.484 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:37.485 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:37.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:37.486 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:37.486+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:37.587 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:37.588+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:37.662 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:37.662 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:37.663 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:37.663 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:37.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:37.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-06-03T05:26:37.688 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:37.689+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:37.792 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:37.791+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:37.807 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:37.807 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:37.807 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:37.807 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:37.807 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:37.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:37.890 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:37.891+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:37.996 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 139.12450808599999, 2024-06-03T05:26:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.006548939999998, 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:38.008 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:38.008 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:37.995+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:38.009 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:38.093 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:38.093+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:38.159 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:38.160 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:38.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:38.193 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:38.193+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:38.295 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:38.295+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:38.364 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:38.365 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:38.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:38.396 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:38.396+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:38.498 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:38.498+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:38.565 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:38.566 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:38.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:38.620 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:38.620+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:38.700 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:38.700+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:38.710 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:38.711 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:38.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:38.805 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:38.805+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:38.897 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:38.897 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:38.902 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:38.902+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:38.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:39.002 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:39.002+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:39.095 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 147.27799528099999, 2024-06-03T05:26:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 147.019163478, 2024-06-03T05:26:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:39.113 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:39.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:39.116 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:39.115+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:39.216 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:39.216+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:39.289 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:39.290 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:39.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:39.341 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:39.341+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:39.418 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:39.418+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:39.475 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:39.475 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:39.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:39.526 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:39.526+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:39.618 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:39.619+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:39.691 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:39.691 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:39.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:39.691 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:39.691 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:39.702 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:39.744 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:39.744+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:39.821 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:39.821+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:39.878 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:39.878 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:39.878 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:39.878 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:39.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:39.930 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:39.929+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:40.022 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:40.022+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:40.062 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:40.062 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:40.062 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:40.063 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:40.063 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:40.071 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:40.156 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:40.156+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:40.224 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:40.224+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:40.249 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 142.583249061, 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.435275320999999, 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:40.259 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:40.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:40.324 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:40.325+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:40.393 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:40.394 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:40.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:40.448 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:40.448+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:40.526 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:40.526+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:40.538 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:40.538 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:40.538 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:40.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:40.632 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:40.632+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:40.728 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:40.728+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:40.775 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:40.775 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 161.90604274899999, 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:40.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:40.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.781 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.731086417, 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.782 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:40.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:40.786 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.730123983, 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:40.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:40.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.792 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.72933398000001, 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.694680119, 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.693990622, 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:40.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.69331727299999, 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.620884158, 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.86289696599999, 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.61424664500001, 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.613001258, 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.61194174900001, 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.57909335400001, 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.57800822300001, 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.57699120500001, 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.55720768800001, 2024-06-03T05:26:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 155.43259969499999, 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 155.43178003099999, 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 155.431079082, 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 149.02402534999999, 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:40.883 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:40.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:40.885 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:40.885+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:40.986 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:40.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:41.061 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:41.061 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:41.061 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:41.061 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:41.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:41.115 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:41.115+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:41.188 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:41.188+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:41.207 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:41.207 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:41.207 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:41.207 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:41.207 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:41.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:41.289 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:41.289+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:41.390 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:41.390+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:41.395 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:41.395 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:41.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:41.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:41.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:41.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:41.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.52378597200001, 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:41.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:41.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:41.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.405826825999998, 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:41.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:41.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:41.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:41.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:41.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:41.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:41.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:41.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:41.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:41.404 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:41.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:41.491 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:41.492+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:41.540 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:41.540 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:41.540 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:41.540 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:41.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:41.592 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:41.593+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:41.692 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:41.693+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:41.723 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:41.723 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:41.723 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:41.723 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:41.723 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:41.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:41.817 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:41.817+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:41.895 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:41.895+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:41.897 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:07.509 51929== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:41.905 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:41.905 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:41.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:41.906 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:41.906 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:41.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:41.996 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:41.996+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:42.088 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:42.088 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:42.088 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:42.088 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:42.097 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:42.097 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:42.097+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:42.116 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:07.728 51929== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:42.197 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:42.197+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:42.259 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:07.856 52797== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:42.270 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:42.271 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:42.271 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:42.271 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:42.271 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:42.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:42.298 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:42.298+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:42.399 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:42.399+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:42.418 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 150.60156353400001, 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:42.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:42.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:42.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:42.423 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 150.34273173099999, 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:42.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:42.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:42.427 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:42.428 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:42.497 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:08.094 52797== Warning: unimplemented fcntl command: 1036 2024-06-03T05:26:42.500 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:42.501+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:42.583 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:42.583 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:42.583 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:42.583 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:42.593 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:42.634 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:42.634+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:42.702 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:42.702+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:42.766 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:42.766 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:42.767 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:42.767 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:42.767 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:42.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:42.820 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:42.820+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:42.904 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:42.904+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:42.909 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:42.910 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:42.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:42.910 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:42.910 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:42.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:43.005 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:43.005+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:43.093 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:43.093 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:43.093 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:43.094 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:43.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:43.106 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:43.106+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:43.221 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:43.221+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:43.237 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:43.237 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:43.237 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:43.237 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:43.237 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:43.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:43.330 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:43.330+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:43.409 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:43.409+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 145.77636624199999, 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:43.443 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:43.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:43.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:43.447 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.628392502000001, 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:43.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:43.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:43.451 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:43.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:43.510 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:43.510+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:43.586 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:43.586 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:43.586 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:43.586 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:43.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:43.636 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:43.636+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:43.712 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:43.712+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:43.768 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:43.768 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:43.768 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:43.768 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:43.768 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:43.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:43.813 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:43.813+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:43.914 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:43.915+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:44.007 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 165.13561493200001, 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 163.96065859999999, 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 163.95969616599999, 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 163.95890616299999, 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 163.92425230200001, 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 163.92356280499999, 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 163.92288945600001, 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 163.85045634100001, 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 163.09246914900001, 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.843818828, 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.84257344100001, 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.841513932, 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.808665537, 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.807580406, 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.080 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.806563388, 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:44.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.78677987099999, 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:44.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:44.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:44.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.66217187800001, 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:44.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:44.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:44.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:44.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:44.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.661352214, 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:44.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:44.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.66065126500001, 2024-06-03T05:26:44.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:44.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:44.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.253597533, 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:44.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:44.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.110 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:44.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:44.112 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:44.112+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:44.213 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:44.213+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:44.246 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:44.246 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:44.247 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:44.247 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:44.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:44.313 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:44.314+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:44.389 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:44.390 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:44.390 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:44.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:44.445 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:44.445+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:44.515 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:44.515+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:44.551 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.67991343400001, 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.561954287999999, 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:44.562 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:44.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:44.616 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:44.616+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:44.700 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:44.700 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:44.700 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:44.700 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:44.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:44.716 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:44.717+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:44.817 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:44.817+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:44.918 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:44.919+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:44.920 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:44.920 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:44.920 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:44.920 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:44.920 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:44.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:45.019 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:45.020+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:45.105 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:45.105 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:45.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:45.105 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:45.105 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:45.114 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:45.158 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:45.158+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:45.222 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:45.222+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:45.251 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:45.252 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:45.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:45.323 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:45.322+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:45.424 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:45.424+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:45.445 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:45.446 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:45.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:45.524 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:45.524+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:45.625 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:45.626+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:45.633 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 153.81649145700001, 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:45.634 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:45.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:45.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 153.55765965399999, 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:45.638 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:45.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:45.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:45.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:45.642 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:45.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:45.726 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:45.726+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:45.827 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:45.827+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:45.844 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:45.844 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:45.844 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:45.844 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:45.853 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:45.938 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:45.937+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:46.031 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:46.032+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:46.047 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:46.047 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:46.047 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:46.047 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:46.047 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:46.058 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:46.142 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:46.142+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:46.231 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:46.232+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:46.253 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:46.253 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:46.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:46.253 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:46.253 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:46.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:46.332 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:46.333+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:46.434 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:46.434+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:46.447 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:46.447 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:46.447 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:46.447 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:46.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:46.534 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:46.535+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:46.633 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:46.633 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:46.633 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:46.633 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:46.634 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:46.635 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:46.635+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:46.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:46.736 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:46.736+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:46.823 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:46.823 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:46.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 149.156838717, 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:46.824 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:46.825 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:46.825 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:46.825 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:46.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.825 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:46.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:46.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:46.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:46.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:46.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:46.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:46.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:46.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:46.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.008864977000002, 2024-06-03T05:26:46.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:46.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:46.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:46.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:46.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:46.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:46.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:46.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:46.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:46.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:46.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:46.834 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:46.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:46.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:46.834 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:46.834 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:46.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:46.837 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:46.838+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:46.939 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:46.939+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:46.973 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:46.974 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:46.974 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:46.974 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:46.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:47.040 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:47.041+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:47.115 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:47.115 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:47.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:47.141 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:47.141+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:47.242 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:47.242+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:47.305 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 168.43682330199999, 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:47.311 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.26186697, 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:47.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.316 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.260904536, 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:47.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.321 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.26011453300001, 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:47.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:47.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.225460672, 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.327 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.224771175, 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:47.332 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:47.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.337 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.22409782599999, 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 167.151664711, 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.343 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:47.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:47.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 166.39367751899999, 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:47.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 166.14502719800001, 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.353 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:47.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:47.358 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 166.143781811, 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:47.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:47.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 166.14272230200001, 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:47.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:47.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 166.10987390700001, 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:47.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:47.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 166.10878877600001, 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:47.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 166.10777175800001, 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.379 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 166.08798824100001, 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:47.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:47.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 161.96338024799999, 2024-06-03T05:26:47.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:47.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:47.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.394 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 161.96256058399999, 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:47.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:47.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 161.961859635, 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.400 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:47.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:47.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:47.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 155.55480590299999, 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:47.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.408 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:47.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:47.410 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:47.410+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:47.511 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:47.511+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:47.543 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:47.543 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:47.544 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:47.544 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:47.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:47.612 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:47.612+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:47.684 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:47.685 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:47.685 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:47.685 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:47.685 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:47.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:47.738 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:47.738+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:47.814 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:47.814+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:47.830 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 148.95937443599999, 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.84141529, 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:47.840 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:47.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:47.914 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:47.915+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:47.985 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:47.985 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:47.985 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:47.985 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:47.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:48.036 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:48.036+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:48.116 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:48.116+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:48.167 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:48.167 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:48.167 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:48.167 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:48.168 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:48.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:48.219 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:48.218+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:48.317 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:48.317+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:48.350 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:48.350 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:48.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:48.350 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:48.350 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:48.359 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:48.444 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:48.444+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:48.519 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:48.519+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:48.532 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:48.532 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:48.532 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:48.532 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:48.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:48.619 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:48.620+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:48.715 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:48.715 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:48.715 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:48.720 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:48.721+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:48.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:48.843 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:48.843+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:48.862 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:48.862 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:48.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:48.862 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:48.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.862 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:48.862 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:48.862 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 157.04476875399999, 2024-06-03T05:26:48.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:48.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:48.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:48.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:48.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:48.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:48.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:48.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:48.866 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:48.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:48.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 156.785936951, 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:48.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:48.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:48.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:48.870 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:48.871 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:48.871 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:48.871 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:48.872 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:48.922 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:48.923+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:49.024 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:49.024+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:49.045 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:49.046 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:49.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:49.141 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:49.141+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:49.226 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:49.226+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:49.229 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:49.230 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:49.230 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:49.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:49.327 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:49.328+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:49.417 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:49.418 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:49.418 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:49.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:49.469 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:49.468+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:49.529 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:49.529+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:49.600 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:49.600 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:49.600 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:49.600 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:49.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:49.651 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:49.651+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:49.731 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:49.731+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:49.783 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:49.783 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:49.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:49.837 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:49.837+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 152.26525250200001, 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:49.932 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:49.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.117278761999998, 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:49.936 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:49.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:49.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:49.940 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:49.941 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:49.933+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:49.942 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:50.033 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:50.033+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:50.088 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:50.088 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:50.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:50.135 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:50.135+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:50.236 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:50.236+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:50.272 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:50.272 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:50.273 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:50.273 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:50.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:50.337 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:50.337+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:50.438 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:50.438+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:50.488 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:50.488 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:50.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:50.488 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 171.59611856199999, 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:50.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.490 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:50.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:50.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:50.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.42116222999999, 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:50.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:50.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:50.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.42019979599999, 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:50.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:50.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.419409793, 2024-06-03T05:26:50.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:50.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:50.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.38475593199999, 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:50.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:50.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.38406643499999, 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.38339308600001, 2024-06-03T05:26:50.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:50.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.31095997099999, 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:50.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:50.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.55297277899999, 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:50.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:50.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.304322458, 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:50.538 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:50.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:50.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.30307707099999, 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:50.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.545 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:50.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:50.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:50.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.302017562, 2024-06-03T05:26:50.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:50.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.554 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.269169167, 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:50.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:50.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.268084036, 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.560 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:50.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:50.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.26706701800001, 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.565 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:50.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 169.247283501, 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:50.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 165.12267550799999, 2024-06-03T05:26:50.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:50.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:50.580 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 165.12185584400001, 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:50.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 165.12115489499999, 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.71410116300001, 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:50.594 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:50.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:50.598 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:50.598+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:50.698 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:50.698+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:50.733 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:50.733 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:50.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:50.799 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:50.799+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:50.877 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:50.877 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:50.877 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:50.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:50.900 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:50.900+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:51.001 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:51.001+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:51.023 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.151841582, 2024-06-03T05:26:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.033882435999999, 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:51.032 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:51.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:51.102 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:51.101+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:51.179 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:51.179 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:51.179 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:51.179 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:51.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:51.231 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:51.230+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:51.303 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:51.302+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:51.366 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:51.367 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:51.367 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:51.367 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:51.367 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:51.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:51.418 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:51.417+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:51.505 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:51.504+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:51.549 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:26:51.550 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:51.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:51.643 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:51.642+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:51.706 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:51.706+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:51.734 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:51.734 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:51.734 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:51.734 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:51.744 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:51.807 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:51.807+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:51.908 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:51.907+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:51.918 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:51.918 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:51.918 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:51.918 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:51.918 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:51.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:52.013 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:52.012+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:52.107 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 160.290047751, 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:26:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 160.03121594800001, 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:26:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:26:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:52.116 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:52.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:52.117 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:52.117+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:52.140 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:26:52.090+0000 11fae640 -1 mon.b@1(leader).osd e43 definitely_dead 0 2024-06-03T05:26:52.218 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:52.218+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:52.240 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:26:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:26:52.292 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:52.292 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:52.292 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:52.292 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:52.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:52.319 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:52.319+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:52.339 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:26:52.339 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:26:52.420 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:52.419+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:52.434 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:26:52.434 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:26:52.477 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:52.477 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:52.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:52.520 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:52.520+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:52.533 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:26:52.533 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:26:52.621 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:52.621+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:52.664 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:52.664 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:52.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:52.722 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:52.722+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:52.786 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:26:52.822 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:52.822+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:52.850 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:52.850 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:52.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:52.882 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:26:52.882 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:26:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:26:52.923 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:52.923+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:53.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:26:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:26:53.007 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:53.007 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:26:53.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:26:53.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:53.059 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:53.058+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:53.124 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:53.125+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:53.192 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:53.192 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:53.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:53.225 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:53.225+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:53.326 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:53.326+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:53.354 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:53.354 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.5. 2024-06-03T05:26:53.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:53.428 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:53.427+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:53.481 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:26:53.495 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-06-03T05:26:53.495 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-03T05:26:53.495 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-06-03T05:26:53.538 INFO:tasks.ceph.mon.a:Started 2024-06-03T05:26:53.538 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-06-03T05:26:53.538 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:26:53.538 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:53.536+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:53.570 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:53.570 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:53.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:53.581 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:26:53.630 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:53.630+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:53.722 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:53.722 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:53.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:53.734 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:53.734+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:53.873 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:53.873+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:53.888 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:53.888 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:53.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:53.933 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:53.932+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:54.034 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:54.033+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:54.039 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:54.039 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:54.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:54.135 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:54.134+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:54.187 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:54.187 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:54.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:54.236 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:54.236+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:54.337 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:54.336+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:54.388 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:54.388 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:54.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:54.445 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:54.443+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:54.456 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:26:54.539 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:54.538+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:54.582 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:54.583 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:54.592 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:54.677 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:54.676+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:54.743 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:54.742+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:54.767 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:54.767 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:54.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:54.861 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:54.860+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:54.943 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:54.943+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:54.951 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:54.951 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:54.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:55.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:26:55.048 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:55.047+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:55.137 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:55.137 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:55.145 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:55.144+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:55.147 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:55.245 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:55.245+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:55.355 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:26:55.356 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:26:55.369 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:55.369+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:55.423 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:55.423 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:55.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:55.548 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:55.547+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:55.556 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:26:55.556 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:26:55.557 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:55.555+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:55.603 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:26:55.651 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:55.651+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:55.751 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:55.751+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:55.809 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:55.809 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:55.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:55.852 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:55.852+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:55.901 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:26:55.901 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:26:55.954 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:55.953+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:56.019 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:26:56.019 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:26:56.055 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:56.055+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:56.058 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:26:56.155 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:56.155+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:56.167 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:26:56.168 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:26:56.170 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:26:56.229 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:56.229 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:56.230 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:56.230 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:56.230 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:56.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:56.281 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:56.279+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:56.292 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:26:56.292 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:26:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:26:56.297 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:26:56.356 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:56.357+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:56.542 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:56.542+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:56.558 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:56.559+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:56.573 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.602746263, 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:26:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:26:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:26:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:26:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:26:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.36737615299999998, 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:26:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:26:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:26:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:56.582 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:56.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:56.660 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:56.659+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:56.718 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:56.718 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:56.718 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:56.718 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:56.728 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:26:56.771 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:56.769+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:56.861 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:56.860+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:56.907 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:26:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:26:56.908 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:56.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:26:57.002 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:57.001+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:57.062 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:57.062+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:57.095 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 159.42929195900001, 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:57.096 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:26:57.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:57.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 39.281318218999999, 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:26:57.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:26:57.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:26:57.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:26:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:26:57.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:26:57.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:26:57.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:26:57.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:57.104 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:57.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:57.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:26:57.104 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:26:57.104 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:57.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:26:57.162 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:57.162+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:57.246 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:57.246 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:57.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:26:57.268 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:57.267+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:57.329 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:26:57.382 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:57.382+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:57.395 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:57.395 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:57.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:26:57.465 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:57.464+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:57.475 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:26:57.566 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:57.566+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:57.632 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 178.76297726799999, 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:26:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:26:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 177.58802093599999, 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:26:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:26:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 177.58705850199999, 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:26:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 177.586268499, 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:26:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:26:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:26:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 177.55161463799999, 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:26:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 177.55092514099999, 2024-06-03T05:26:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:26:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:26:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:26:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:26:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:26:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:26:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:26:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:26:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:26:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 177.55025179200001, 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:26:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:26:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 177.47781867699999, 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:26:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:26:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 176.71983148499999, 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:26:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:26:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 176.471181164, 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:26:57.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:26:57.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 176.46993577699999, 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.690 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:26:57.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:57.695 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 176.468876268, 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:26:57.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:26:57.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 176.436027873, 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.701 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:26:57.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:26:57.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 176.434942742, 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:26:57.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:26:57.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.711 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 176.43392572400001, 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.716 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 176.414142207, 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:26:57.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:26:57.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:26:57.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 172.28953421400001, 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.722 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:26:57.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 172.28871455000001, 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.727 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 172.28801360099999, 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.732 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:26:57.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:26:57.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 165.88095986900001, 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:26:57.737 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:57.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:26:57.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:26:57.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:26:57.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.741 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:57.741 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:57.741 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:57.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:26:57.743 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:57.742+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:57.844 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:57.843+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:57.918 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:26:57.918 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:57.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:26:57.972 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:57.972+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:58.045 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:58.045+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:58.060 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:58.060 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:26:58.061 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:26:58.061 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:26:58.061 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:58.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:26:58.147 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:58.146+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:58.151 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:26:58.248 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:26:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:26:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:26:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:26:58.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.37705696, 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.249 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:26:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:26:58.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.259097814, 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:26:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:26:58.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:26:58.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:26:58.257 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:26:58.257 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:58.248+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:58.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:26:58.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:26:58.350 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:26:58.349+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:26:58.374 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [6, 4] live_osds: [0, 1, 2, 3, 5, 7] 2024-06-03T05:26:58.374 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:26:58.375 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-06-03T05:26:58.375 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-03T05:26:58.375 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-03T05:26:58.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-06-03T05:26:58.414 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:58.414 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:26:58.414 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:26:58.415 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:58.424 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:26:58.450 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:58.450+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:58.540 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:26:58.540 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-03T05:26:58.540 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:26:58.551 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:58.551+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:58.652 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:58.652+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:58.653 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:26:58.653 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:26:58.654 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:58.654 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:58.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:26:58.748 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:26:58.749 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:26:58.752 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:58.753+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:58.799 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:58.799 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:58.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:26:58.853 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:58.853+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:58.864 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:26:58.865 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:26:58.943 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:58.943 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:58.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:26:58.977 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:26:58.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:26:58.978 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:58.977+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:58.981 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:26:59.055 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:59.054+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:59.091 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:26:59.092 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:26:59.094 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:26:59.113 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:59.113 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:59.122 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:26:59.164 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:59.163+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:59.200 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:26:59.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:26:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:26:59.204 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:26:59.257 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:26:59.257+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:26:59.296 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:59.296 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:59.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:26:59.344 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:26:59.393 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:59.392+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:59.440 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:59.441 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:59.450 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:26:59.457 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:59.458+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:59.558 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:26:59.558+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:26:59.626 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:59.626 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:59.636 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:26:59.679 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:59.677+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:59.760 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:26:59.759+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:26:59.812 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:59.812 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:26:59.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:26:59.860 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:26:59.861+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:26:59.962 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:26:59.961+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:26:59.999 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:26:59.999 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:00.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:00.093 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:00.092+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:00.163 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:00.163+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:00.183 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:00.183 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:00.193 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:00.278 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:00.277+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:00.365 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:00.365+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:00.368 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:00.368 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:00.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:00.466 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:00.466+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:00.512 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:00.512 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:00.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:00.567 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:00.566+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:00.668 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:00.668+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:00.693 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:00.694 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:00.703 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:00.772 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:00.771+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:00.866 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:27:00.871 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:00.871+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:00.879 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:00.879 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:00.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:00.973 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:00.973+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:01.022 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:01.022 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:01.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:01.073 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:01.073+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:01.165 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:27:01.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:27:01.206 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:01.206 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:01.211 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:01.209+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:01.215 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:01.276 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:01.275+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:01.363 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:01.363 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:01.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:01.417 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:01.416+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:01.477 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:01.476+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:01.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:27:01.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:27:01.578 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:01.577+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:01.578 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:01.578 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:01.579 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:01.579 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:01.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:01.617 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:08.005 54353== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:01.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:27:01.635 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:27:01.679 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:01.678+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:01.755 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:27:01.755 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:27:01.765 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:01.765 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:01.765 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:01.765 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:01.765 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:01.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:27:01.855 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:27:01.855 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:01.854+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:01.858 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:08.246 54353== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:01.858 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:27:01.880 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:01.879+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:01.975 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:27:01.975 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:27:01.976 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:27:01.977 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:01.977 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:27:01.989 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:01.989 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:27:02.001 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:02.000+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:02.077 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:27:02.077 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:27:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:27:02.082 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:27:02.083 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:02.082+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:02.129 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:02.130 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:02.130 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:02.130 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:02.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:02.183 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:02.182+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:02.272 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:02.273 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:02.273 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:02.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:02.324 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:02.323+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:02.385 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:02.386+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:02.464 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 170.64271423700001, 2024-06-03T05:27:02.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 170.38388243399999, 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:02.469 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:27:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:27:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:27:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:27:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:27:02.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:27:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:27:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:27:02.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:27:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:27:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:27:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:27:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:27:02.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:27:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:27:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:02.476 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.3119704790000002, 2024-06-03T05:27:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:27:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:02.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:27:02.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:27:02.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.482 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.2218272270000003, 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:27:02.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:02.486 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:02.488 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:02.488 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:02.487+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:02.606 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:02.605+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:02.621 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:02.622 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:02.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:02.715 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:02.714+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:02.790 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:02.790+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:02.805 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:02.806 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:02.806 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:02.806 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:02.806 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:02.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:02.880 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:27:02.900 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:02.899+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:02.992 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:02.991+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:03.009 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:03.009 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.0435564890000002, 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:27:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.8081863790000003, 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:03.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:03.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:03.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:27:03.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:27:03.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.019 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:03.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.019 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:03.019 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:03.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:03.105 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:03.104+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:03.193 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:03.193+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:03.199 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:03.199 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:03.199 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:03.199 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:03.209 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:03.294 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:03.293+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:03.384 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:03.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:03.385 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:03.385 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:03.385 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:03.395 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:03.396+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:03.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:03.496 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:03.497+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:03.552 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [6, 4] live_osds: [0, 1, 2, 3, 5, 7] 2024-06-03T05:27:03.552 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:27:03.577 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 165.911147634, 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:27:03.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.763173893999998, 2024-06-03T05:27:03.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:03.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:03.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:03.586 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:03.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:03.597 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:03.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:03.698 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:03.698+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:03.721 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:03.721 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:03.721 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:27:03.721 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:03.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:03.801 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:03.798+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:03.862 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:03.863 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:03.863 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:27:03.863 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:27:03.863 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:03.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:03.900 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:03.900+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:04.001 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:04.001+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:04.056 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:04.056 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:27:04.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:27:04.056 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 185.18699439299999, 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.057 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:27:04.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:27:04.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:27:04.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:27:04.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:27:04.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 184.012038061, 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:27:04.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:27:04.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 184.011075627, 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:04.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:27:04.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:27:04.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.074 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 184.01028562400001, 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:27:04.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:27:04.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.080 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 183.975631763, 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:27:04.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.082 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:04.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:27:04.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.086 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 183.974942266, 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 183.97426891699999, 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:27:04.092 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 183.90183580199999, 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.098 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.103 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 183.14384860999999, 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:27:04.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:27:04.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 182.89519828900001, 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:27:04.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 182.893952902, 2024-06-03T05:27:04.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:04.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:27:04.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.119 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 182.89289339300001, 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 182.86004499800001, 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:27:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:27:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:27:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 182.85895986700001, 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:27:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 182.85794284900001, 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:27:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:27:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 182.838159332, 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:27:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:27:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 178.71355133899999, 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:27:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 178.71273167499999, 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:27:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:27:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 178.71203072599999, 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:27:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 172.30497699399999, 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:27:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:27:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:27:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:27:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:27:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.168 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:04.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:27:04.170 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:27:04.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:04.171 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:04.169+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:04.270 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:04.270+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:04.345 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:27:04.346 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:04.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:04.399 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:04.399+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:04.472 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:04.471+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:04.494 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:04.494 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:27:04.495 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:27:04.495 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:04.499 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:27:04.499 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:27:04.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:04.589 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:04.589+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:04.592 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:27:04.592 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:27:04.674 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:04.673+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 165.809722652, 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.691763506000001, 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:27:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:27:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:27:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:04.690 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:04.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:04.697 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:27:04.697 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:27:04.778 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:04.775+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:04.795 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:27:04.796 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:27:04.801 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:27:04.831 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:04.831 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:04.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:04.884 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:04.883+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:04.902 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:27:04.902 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:27:04.904 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:27:04.978 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:04.977+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:04.998 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:27:04.998 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:27:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:27:05.002 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:27:05.006 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:27:05.021 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:05.021 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:05.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:05.116 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:05.115+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:05.179 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:05.179+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:05.226 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:05.226 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:05.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:05.280 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:05.280+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:05.380 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:05.380+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:05.420 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:05.420 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:05.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:05.514 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:05.513+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:05.583 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:05.582+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:05.608 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:05.608 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:05.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:05.703 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:05.702+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:05.777 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:27:05.785 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:05.785+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:05.791 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-06-03T05:27:05.791 INFO:tasks.mon_recovery:stopping all monitors 2024-06-03T05:27:05.791 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-03T05:27:05.792 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:27:05.800 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:05.800 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:05.806 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:27:05.796+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 51927) UID: 0 2024-06-03T05:27:05.808 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:27:05.797+0000 9f9e640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-06-03T05:27:05.810 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:05.885 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:05.886+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:05.987 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:05.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:06.022 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:06.022 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:06.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:06.130 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:06.130+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:06.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:27:06.221 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:06.221 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:06.221 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:06.220+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:06.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:06.289 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:06.290+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:06.385 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:06.385 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:06.390 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:06.390+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:06.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:06.492 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:06.490+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:06.537 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:32.149 51929== 2024-06-03T05:27:06.537 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:32.149 51929== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:27:06.548 INFO:tasks.ceph.mon.b.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:27:06.585 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:06.585 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:06.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:06.597 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:06.597+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:06.692 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:06.693+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:06.770 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:06.770 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:06.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:06.823 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:06.822+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:06.894 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:06.894+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:06.913 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:06.922 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:06.994 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:06.994+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:07.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:27:07.060 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:07.060 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:07.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:07.095 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:07.095+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:07.197 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:07.197+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:07.202 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:07.202 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:07.211 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:07.298 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:07.297+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:07.343 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:07.344 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:07.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:07.398 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:07.398+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:07.487 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:07.487 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:07.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:07.499 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:07.499+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:07.600 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:07.600+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:07.630 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:07.630 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:07.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:07.701 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:07.701+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:07.772 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:07.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:07.826 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:07.825+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:07.902 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:07.902+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:07.925 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:07.926 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:07.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-06-03T05:27:07.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:08.020 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:08.019+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:08.104 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:08.103+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:08.113 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:08.113 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:08.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:08.208 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:08.208+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:08.296 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:08.296 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:08.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:08.305 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:08.304+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:08.406 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:08.406+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:08.482 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:08.482 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:08.491 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:08.533 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:08.533+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:08.553 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [6, 4] live_osds: [0, 1, 2, 3, 5, 7] 2024-06-03T05:27:08.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:27:08.553 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-06-03T05:27:08.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-06-03T05:27:08.608 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:08.609+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:08.666 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:08.667 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:08.676 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:08.720 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:08.719+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:08.810 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:08.810 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:08.810 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:08.811+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:08.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:08.912 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:08.912+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:08.994 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:08.994 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:09.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:09.013 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:09.013+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:09.114 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:09.113+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:09.136 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:09.136 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:09.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:09.232 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:09.230+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:09.316 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:09.316+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:09.320 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:09.320 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:09.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:09.417 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:09.417+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:09.504 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:09.504 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:09.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:09.554 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:09.554+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:09.618 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:09.619+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:09.685 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:09.686 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:09.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:09.739 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:09.739+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:09.821 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:09.821+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:09.832 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:09.832 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:09.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:09.922 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:09.922+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:09.987 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:09.988 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:09.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:10.022 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:10.023+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:10.123 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:10.124+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:10.131 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:10.132 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:10.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:10.225 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:10.225+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:10.273 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:10.274 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:10.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:10.325 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:10.326+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:10.426 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:10.426+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:10.457 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:10.457 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:10.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:10.527 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:10.528+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:10.600 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:10.600 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:10.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:10.653 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:10.653+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:10.730 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:10.730+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:10.742 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:10.742 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:10.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:10.831 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:10.831+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:10.897 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:10.897 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:10.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:10.950 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:10.949+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:11.033 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:11.033+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:11.079 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:11.080 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:11.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:11.133 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:11.134+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:11.248 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:11.248+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:11.265 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:11.265 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:11.274 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:11.359 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:11.359+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:11.437 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:11.437+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:11.447 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:11.448 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:11.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:11.541 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:11.541+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:11.630 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:11.631 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:11.638 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:11.638+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:11.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:11.739 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:11.739+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:11.818 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:11.818 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:11.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:11.870 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:11.869+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:11.893 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:27:11.894 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-06-03T05:27:11.894 INFO:tasks.ceph.mon.b:Stopped 2024-06-03T05:27:11.894 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-03T05:27:11.895 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:27:11.940 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:11.941+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:12.019 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:12.019 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:12.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:12.042 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:27:12.030+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 52792) UID: 0 2024-06-03T05:27:12.042 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:27:12.032+0000 9f9e640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-06-03T05:27:12.043 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:12.043+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:12.143 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:12.143+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:12.201 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:12.201 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:12.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:12.256 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:12.255+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:12.344 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:12.345+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:12.390 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:12.390 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:12.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:12.447 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:12.447+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:12.546 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:12.547+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:12.566 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:12.566 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:12.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:12.663 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:12.663+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:12.749 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:12.749+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:12.773 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:12.892 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:12.892+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:12.917 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:38.515 52797== 2024-06-03T05:27:12.918 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:38.515 52797== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:27:12.932 INFO:tasks.ceph.mon.c.smithi038.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:27:13.007 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:12.958+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:13.008 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:13.008 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:13.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:13.051 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:13.051+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:13.152 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:13.153+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:13.160 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:13.160 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:13.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:13.253 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:13.253+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:13.301 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:13.302 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:13.310 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:13.354 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:13.354+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:13.443 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:13.443 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:13.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:13.454 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:13.454+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:13.555 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:13.555+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:13.588 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:13.588 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:13.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:13.656 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:13.656+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:13.731 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:13.732 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:13.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:13.784 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:13.784+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:13.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-06-03T05:27:13.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-06-03T05:27:13.857 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:13.858+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:13.875 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:13.876 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:13.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:13.958 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:13.958+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:14.024 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:14.024 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:14.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:14.080 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:14.080+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:14.161 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:14.161+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:14.169 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:14.169 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:14.178 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:14.262 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:14.262+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:14.356 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:14.356 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:14.363 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:14.363+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:14.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:14.483 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:14.484+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:14.498 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:14.498 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:14.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:14.592 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:14.592+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:14.666 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:14.666+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:14.680 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:14.680 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:14.690 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:14.767 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:14.768+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:14.865 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:14.865 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:14.868 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:14.868+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:14.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:14.969 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:14.969+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:15.048 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:15.048 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:15.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:15.099 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:15.099+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:15.171 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:15.171+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:15.230 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:15.230 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:15.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:15.283 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:15.283+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:15.374 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:15.374 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:15.375 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:15.376+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:15.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:15.478 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:15.479+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:15.557 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:15.558 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:15.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:15.579 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:15.580+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:15.683 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:15.683+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:15.740 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:15.740 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:15.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:15.791 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:15.790+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:15.884 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:15.885+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:15.924 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:15.924 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:15.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:15.986 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:15.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:16.080 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:16.081 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:16.086 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:16.086+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:16.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:16.187 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:16.187+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:16.226 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:16.226 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:16.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:16.288 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:16.288+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:16.389 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:16.390+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:16.408 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:16.409 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:16.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:16.490 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:16.490+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:16.551 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:16.551 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:16.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:16.591 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:16.591+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:16.692 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:16.693+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:16.736 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:16.736 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:16.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:16.795 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:16.795+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:16.905 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:16.905+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:16.919 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:16.919 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:16.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:16.997 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:16.997+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:17.077 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:17.077 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:17.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:17.128 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:17.128+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:17.198 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:17.199+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:17.260 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:17.260 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:17.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:17.314 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:17.315+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:17.400 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:17.400+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:17.403 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:17.403 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:17.412 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:17.501 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:17.501+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:17.586 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:17.586 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:17.597 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:17.602 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:17.602+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:17.702 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:17.703+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:17.770 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:17.770 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:17.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:17.822 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:17.822+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:17.904 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:17.904+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:17.965 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:17.965 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:17.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:27:17.996 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-06-03T05:27:17.996 INFO:tasks.ceph.mon.c:Stopped 2024-06-03T05:27:17.996 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-03T05:27:17.996 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:27:18.018 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:18.017+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:18.031 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-06-03T05:27:18.021+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 54347) UID: 0 2024-06-03T05:27:18.032 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-06-03T05:27:18.022+0000 9f9e640 -1 mon.a@0(probing) e1 *** Got Signal Terminated *** 2024-06-03T05:27:18.106 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:18.106+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:18.155 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:18.156 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:18.165 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:18.206 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:18.207+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:18.307 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:18.307+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:18.341 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:18.341 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:18.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:18.435 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:18.435+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:18.510 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:18.510+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:18.524 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:18.524 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:18.538 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:18.623 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:18.623+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:18.711 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:18.712+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:18.716 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:18.717 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:18.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:18.812 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:18.813+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:18.816 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:25.205 54353== 2024-06-03T05:27:18.816 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:25.205 54353== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:27:18.827 INFO:tasks.ceph.mon.a.smithi038.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:27:18.903 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:18.903 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:18.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:18.917 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:18.917+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:19.017 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:19.018+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:19.049 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:19.049 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:19.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:19.118 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:19.119+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:19.204 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:19.204 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:19.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:19.219 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:19.220+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:19.320 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:19.320+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:19.346 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:19.346 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:19.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:19.421 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:19.421+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:19.489 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:19.489 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:19.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:19.522 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:19.522+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:19.623 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:19.623+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:19.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-06-03T05:27:19.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-06-03T05:27:19.673 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:19.673 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:19.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:19.728 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:19.728+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:19.816 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:19.816 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:19.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:19.825 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:19.824+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:19.926 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:19.926+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:20.001 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:20.002 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:20.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:20.056 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:20.056+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:20.128 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:20.128+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:20.162 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:20.162 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:20.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:20.229 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:20.229+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:20.329 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:20.329+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:20.345 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:20.345 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:20.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:20.430 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:20.430+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:20.489 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:20.489 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:20.498 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:20.540 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:20.540+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:20.631 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:20.631+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:20.674 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:20.674 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:20.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:20.733 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:20.733+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:20.815 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:20.815 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:20.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:20.866 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:20.866+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:20.934 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:20.935+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:20.997 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:20.998 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:21.007 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:21.049 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:21.049+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:21.136 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:21.136+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:21.183 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:21.183 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:21.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:21.277 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:21.277+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:21.338 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:21.338+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:21.367 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:21.367 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:21.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:21.447 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:21.447+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:21.547 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:21.547+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:21.581 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:21.581 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:21.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:21.675 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:21.675+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:21.751 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:21.750+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:21.766 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:21.766 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:21.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:21.860 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:21.860+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:21.950 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:21.951+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:21.951 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:21.952 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:21.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:22.052 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:22.052+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:22.136 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:22.136 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:22.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:22.189 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:22.188+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:22.253 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:22.253+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:22.291 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:22.291 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:22.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:22.354 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:22.354+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:22.461 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:22.460+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:22.475 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:22.475 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:22.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:22.555 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:22.555+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:22.656 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:22.655+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:22.658 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:22.658 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:22.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:22.756 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:22.756+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:22.842 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:22.842 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:22.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:22.857 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:22.857+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:22.958 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:22.957+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:23.025 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:23.025 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:23.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:23.058 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:23.059+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:23.160 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:23.160+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:23.208 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:23.208 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:23.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:23.263 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:23.263+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:23.361 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:23.361+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:23.365 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:23.365 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:23.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:23.462 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:23.462+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:23.549 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:23.549 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:23.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:23.600 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:23.600+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:23.663 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:23.663+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:23.732 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:23.732 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:23.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:23.765 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:23.764+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:23.865 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:23.865+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:23.875 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:23.875 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:23.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:23.967 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:23.967+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:24.020 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:24.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:24.071 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:24.071+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:27:24.098 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-06-03T05:27:24.099 INFO:tasks.ceph.mon.a:Stopped 2024-06-03T05:27:24.099 INFO:tasks.mon_recovery:forming a minimal quorum for ['b', 'c', 'a'], then adding monitors 2024-06-03T05:27:24.099 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-03T05:27:24.099 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-06-03T05:27:24.102 INFO:tasks.ceph.mon.b:Started 2024-06-03T05:27:24.102 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-03T05:27:24.102 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-06-03T05:27:24.115 INFO:tasks.ceph.mon.c:Started 2024-06-03T05:27:24.115 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-06-03T05:27:24.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:27:24.168 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:24.167+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:24.219 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:24.220 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:24.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:24.276 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:24.276+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:24.365 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:24.366 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:24.372 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:24.373+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:24.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:24.475 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:24.475+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:24.549 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:24.549 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:24.560 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:24.601 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:24.601+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:24.679 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:24.679+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:24.733 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:24.733 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:24.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:24.780 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:24.779+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:24.880 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:24.881+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:24.921 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:24.921 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:24.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:25.016 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:25.016+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:25.082 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:25.082+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:25.109 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:25.109 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:25.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:25.182 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:25.182+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:25.284 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:25.283+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:25.296 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:25.296 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:25.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:25.384 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:25.385+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:25.446 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:25.446 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:25.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:25.485 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:25.486+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:25.587 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:25.587+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:25.593 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:25.593 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:25.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:25.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-06-03T05:27:25.687 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:25.688+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:25.739 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:25.739 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:25.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:25.792 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:25.791+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:25.882 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:25.882 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:25.889 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:25.888+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:25.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:25.990 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:25.990+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:26.067 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:26.067 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:26.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:26.091 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:26.091+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:26.192 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:26.192+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:26.209 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:26.209 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:26.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:26.294 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:26.294+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:26.352 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:26.352 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:26.362 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:26.394 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:26.393+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:26.495 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:26.496+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:26.539 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:26.539 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:26.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:26.598 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:26.598+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:26.714 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:26.714+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:26.729 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:26.729 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:26.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:26.800 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:26.798+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:26.902 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:26.900+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:26.916 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:26.916 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:26.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:27.004 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:27.004+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:27.066 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:27.066 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:27.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:27.120 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:27.120+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:27.205 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:27.204+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:27.251 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:27.252 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:27.262 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:27.306 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:27.306+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:27.398 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:27.398 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:27.407 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:27.406+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:27.408 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:27.508 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:27.508+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:27.586 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:27.587 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:27.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:27.638 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:27.638+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:27.710 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:27.711+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:27.772 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:27.772 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:27.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:27.811 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:27.810+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:27.912 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:27.913+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:27.957 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:27.958 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:27.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:28.053 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:28.053+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:28.129 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:28.129+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:28.144 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:28.144 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:28.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:28.216 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:28.215+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:28.317 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:28.316+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:28.330 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:28.330 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:28.339 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:28.417 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:28.418+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:28.474 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:28.475 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:28.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:28.518 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:28.519+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:28.619 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:28.619+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:28.620 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:28.620 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:28.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:28.721 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:28.720+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:28.815 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:28.815 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:28.821 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:28.822+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:28.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:28.922 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:28.922+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:29.002 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:29.002 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:29.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:29.055 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:29.054+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:29.130 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:29.129+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:29.144 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:29.144 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:29.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:29.225 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:29.225+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:29.326 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:29.326+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:29.330 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:29.330 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:29.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:29.427 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:29.426+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:29.474 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:29.475 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:29.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:29.528 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:29.528+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:29.617 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:29.617 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:29.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:29.669 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:29.668+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:29.730 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:29.730+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:29.802 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:29.802 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:29.812 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:29.854 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:29.854+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:29.932 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:29.932+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:29.990 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:29.990 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:29.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:30.038 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:30.037+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:30.134 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:30.134+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:30.169 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:30.169 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:30.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:30.263 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:30.263+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:30.336 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:30.336+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:30.352 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:30.352 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:30.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:30.446 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:30.446+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:30.538 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:30.538+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:30.541 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:30.541 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:30.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:30.638 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:30.639+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:30.725 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:30.725 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:30.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:30.778 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:30.777+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:30.841 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:30.841+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:30.909 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:30.909 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:30.918 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:30.959 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:30.959+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:31.042 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:31.043+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:31.092 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:31.092 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:31.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:31.145 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:31.144+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:31.244 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:31.244+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:31.276 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:31.276 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:31.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:31.345 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:31.345+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:31.419 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:31.420 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:31.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:31.471 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:31.471+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:31.548 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:31.548+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:31.581 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:31.581 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:31.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:31.649 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:31.649+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:31.726 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:31.727 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:31.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:31.777 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:31.777+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:31.817 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:07.644 53808== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:31.851 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:31.851+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:31.908 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:07.730 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:31.909 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:31.909 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:31.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:31.952 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:31.952+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:32.032 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:07.860 53808== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:32.052 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:32.052+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:32.093 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:32.093 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:32.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:32.124 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:07.946 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:32.153 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:32.153+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:32.236 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:32.236 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:32.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:32.289 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:32.289+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:32.356 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:32.355+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:32.379 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:32.379 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:32.389 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:32.456 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:32.456+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:32.524 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:32.524 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:32.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:32.578 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:32.578+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:32.658 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:32.659+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:32.666 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:32.666 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:32.675 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:32.759 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:32.759+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:32.860 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:32.860+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:32.863 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:32.863 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:32.872 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:32.962 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:32.962+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:33.046 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:33.046 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:33.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:33.062 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:33.062+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:33.164 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:33.164+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:33.229 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:33.229 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:33.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:33.264 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:33.265+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:33.365 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:33.365+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:33.414 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:33.414 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:33.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:33.469 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:33.469+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:33.559 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:33.559 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:33.567 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:33.568+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:33.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:33.668 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:33.669+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:33.742 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:33.742 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:33.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:33.794 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:33.793+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:33.871 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:33.871+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:33.940 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:33.940 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:33.950 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:33.995 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:33.995+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:34.073 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:34.073+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:34.083 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:34.083 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:34.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:34.178 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:34.178+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:34.268 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:34.268 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:34.275 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:34.276+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:34.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:34.377 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:34.377+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:34.458 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:34.458 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:34.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:34.510 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:34.510+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:34.579 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:34.580+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:34.601 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:34.601 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:34.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:34.680 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:34.680+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:34.754 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:34.754 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:34.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:34.808 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:34.808+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:34.882 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:34.882+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:34.896 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:34.896 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:34.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:34.991 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:34.991+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:35.083 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:35.084 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:35.084 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:35.085+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:35.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:35.185 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:35.185+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:35.276 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:35.276 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:35.286 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:35.286+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:35.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:35.387 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:35.387+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:35.463 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:35.463 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:35.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:35.518 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:35.519+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:35.588 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:35.589+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:35.617 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:35.617 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:35.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:35.711 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:35.711+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:35.790 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:35.790+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:35.812 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:35.812 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:35.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:35.891 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:35.891+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:35.992 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:35.993+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:36.003 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:36.003 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:36.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:36.093 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:36.093+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:36.186 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:36.186 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:36.194 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:36.194+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:36.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:36.295 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:36.295+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:36.370 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:36.370 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:36.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:36.396 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:36.396+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:36.497 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:36.497+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:36.553 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:36.553 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:36.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:36.604 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:36.604+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:36.698 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:36.698+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:36.738 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:36.738 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:36.747 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:36.832 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:36.832+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:36.899 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:36.899+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:36.921 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:36.921 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:36.930 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:37.000 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:37.000+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:37.101 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:37.101+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:37.103 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:37.103 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:37.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:37.202 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:37.202+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:37.287 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:37.287 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:37.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:37.339 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:37.338+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:37.403 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:37.404+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:37.472 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:37.472 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:37.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:37.522 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:37.522+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:37.606 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:37.606+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:37.654 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:37.654 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:37.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:37.707 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:37.707+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:37.808 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:37.808+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:37.810 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:37.810 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:37.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:37.909 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:37.910+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:37.953 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:37.953 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:37.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:38.011 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:38.011+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:38.095 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:38.095 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:38.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:38.111 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:38.112+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:38.213 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:38.213+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:38.280 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:38.280 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:38.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:38.314 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:38.314+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:38.415 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:38.415+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:38.463 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:38.463 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:38.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:38.515 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:38.516+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:38.606 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:38.606 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:38.616 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:38.616 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:38.616+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:38.718 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:38.718+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:38.762 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:38.762 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:38.772 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:38.819 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:38.819+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:38.919 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:38.920+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:38.955 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:38.956 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:38.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:39.049 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:39.049+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:39.121 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:39.122+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:39.138 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:39.138 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:39.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:39.222 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:39.222+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:39.321 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:39.321 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:39.323 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:39.323+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:39.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:39.423 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:39.423+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:39.503 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:39.504 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:39.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:39.524 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:39.525+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:39.625 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:39.626+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:39.646 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:39.646 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:39.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:39.726 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:39.727+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:39.799 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:39.799 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:39.810 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:39.851 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:39.851+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:39.929 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:39.929+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:39.986 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:39.987 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:39.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:40.040 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:40.040+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:40.130 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:40.131+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:40.133 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:40.133 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:40.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:40.232 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:40.232+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:40.318 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:40.319 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:40.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:40.372 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:40.372+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:40.434 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:40.434+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:40.460 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:40.460 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:40.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:40.554 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:40.554+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:40.636 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:40.636+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:40.647 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:40.647 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:40.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:40.737 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:40.737+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:40.801 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:40.801 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:40.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:40.838 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:40.838+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:40.939 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:40.939+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:40.983 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:40.983 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:40.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:41.039 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:41.040+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:41.132 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:41.133 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:41.141 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:41.141+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:41.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:41.242 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:41.242+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:41.283 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:41.283 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:41.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:41.343 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:41.343+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:41.426 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:41.427 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:41.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:41.479 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:41.478+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:41.545 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:41.545+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:41.585 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:27:41.585 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:41.585 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 50, 2024-06-03T05:27:41.585 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "ebc09f7e-f094-4421-826f-479fc526fc62", 2024-06-03T05:27:41.585 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-06-03T05:23:10.198378+0000", 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-06-03T05:27:04.038630+0000", 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-06-03T05:26:03.427745+0000", 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-06-03T05:26:52.558950+0000", 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5804032, 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-06-03T05:27:41.586 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 6, 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 2, 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "jewel", 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.587 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-06-03T05:23:56.753149+0000", 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 1, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-03T05:27:41.588 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-06-03T05:27:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "16", 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-06-03T05:27:41.590 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-06-03T05:27:41.591 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-06-03T05:27:41.592 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 8, 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 8, 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.25, 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2, 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2, 2024-06-03T05:27:41.593 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "rbd", 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-06-03T05:24:12.898324+0000", 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 1, 2024-06-03T05:27:41.594 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 8, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 8, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 8, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 8, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 8, 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-06-03T05:27:41.595 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "24", 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-06-03T05:27:41.596 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-06-03T05:27:41.597 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-06-03T05:27:41.598 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 5, 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 5, 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.5, 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.5, 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.5, 2024-06-03T05:27:41.599 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "32b256d9-eee7-435b-8aec-fef2ba90366c", 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:41.600 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 16, 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 39, 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:27:41.603 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/905141127", 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6811/905141127", 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/905141127", 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6813/905141127", 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.604 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "371c2e00-54db-430b-8497-a4c9ff6753b9", 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0.9170379638671875, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 44, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:27:41.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:41.607 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6810/3490615740", 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6812/3490615740", 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6815/3490615740", 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6814/3490615740", 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:41.608 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "e8145335-ec5d-4dd1-9250-a4ab5b159d29", 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:27:41.609 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 46, 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:27:41.611 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/3359326199", 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/3359326199", 2024-06-03T05:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/3359326199", 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/3359326199", 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c", 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:41.613 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 19, 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 45, 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.615 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.616 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/2289724788", 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/2289724788", 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/2289724788", 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/2289724788", 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "46b46f54-04d0-46c0-809c-64fa385e6780", 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 0, 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:41.617 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 39, 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6808", 2024-06-03T05:27:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6809", 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:41.619 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6811", 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.620 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6810", 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6808/3833397294", 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6809/3833397294", 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6811/3833397294", 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6810/3833397294", 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: "exists" 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.621 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f7a50ac0-eba9-4602-b90a-2dbc4bb20593", 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 36, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:41.622 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6800", 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6801", 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:27:41.623 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6803", 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6802", 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6800/3915827095", 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6801/3915827095", 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6803/3915827095", 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6802/3915827095", 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:41.625 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "081024c0-2482-4846-a7bd-594a58f2ceef", 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 0, 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-06-03T05:27:41.626 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 0, 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 36, 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6804", 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6805", 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6807", 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.628 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6806", 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6804/1144336651", 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6805/1144336651", 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1144336651", 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6806/1144336651", 2024-06-03T05:27:41.629 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "exists" 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "6234c757-8f7b-497e-9347-4a7983df592c", 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:41.630 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 45, 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6812", 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:41.631 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6813", 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6815", 2024-06-03T05:27:41.632 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6814", 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.633 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6812/370645108", 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6813/370645108", 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6815/370645108", 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6814/370645108", 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-06-03T05:27:41.634 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:56.081253+0000", 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:41.635 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:55.642638+0000", 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:55.131979+0000", 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:41.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:55.147738+0000", 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:41.637 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.907705+0000", 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 38 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-06-03T05:27:41.638 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.533669+0000", 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.310102+0000", 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 36 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.959957+0000", 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [], 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-06-03T05:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [ 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "2.0", 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: 5 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "2.1", 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: 5 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "2.4", 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: 2, 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: 5 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": {}, 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-06-03T05:27:41.642 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-06-03T05:27:41.643 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-06-03T05:27:41.644 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-06-03T05:27:41.644 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-06-03T05:27:41.644 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-06-03T05:27:41.644 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:41.644 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:41.646 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:41.647 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:41.647 INFO:tasks.thrashosds.thrasher:j is {'epoch': 50, 'fsid': 'ebc09f7e-f094-4421-826f-479fc526fc62', 'created': '2024-06-03T05:23:10.198378+0000', 'modified': '2024-06-03T05:27:04.038630+0000', 'last_up_change': '2024-06-03T05:26:03.427745+0000', 'last_in_change': '2024-06-03T05:26:52.558950+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-03T05:23:56.753149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-06-03T05:24:12.898324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5, 'score_stable': 5, 'optimal_score': 0.5, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '32b256d9-eee7-435b-8aec-fef2ba90366c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 39, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6809', 'nonce': 905141127}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6811', 'nonce': 905141127}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6816', 'nonce': 905141127}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6813', 'nonce': 905141127}]}, 'public_addr': '172.21.15.38:6809/905141127', 'cluster_addr': '172.21.15.38:6811/905141127', 'heartbeat_back_addr': '172.21.15.38:6816/905141127', 'heartbeat_front_addr': '172.21.15.38:6813/905141127', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '371c2e00-54db-430b-8497-a4c9ff6753b9', 'up': 1, 'in': 1, 'weight': 0.9170379638671875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 44, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6810', 'nonce': 3490615740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6812', 'nonce': 3490615740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6815', 'nonce': 3490615740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6814', 'nonce': 3490615740}]}, 'public_addr': '172.21.15.38:6810/3490615740', 'cluster_addr': '172.21.15.38:6812/3490615740', 'heartbeat_back_addr': '172.21.15.38:6815/3490615740', 'heartbeat_front_addr': '172.21.15.38:6814/3490615740', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8145335-ec5d-4dd1-9250-a4ab5b159d29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6805', 'nonce': 3359326199}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6806', 'nonce': 3359326199}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6808', 'nonce': 3359326199}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6807', 'nonce': 3359326199}]}, 'public_addr': '172.21.15.38:6805/3359326199', 'cluster_addr': '172.21.15.38:6806/3359326199', 'heartbeat_back_addr': '172.21.15.38:6808/3359326199', 'heartbeat_front_addr': '172.21.15.38:6807/3359326199', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6801', 'nonce': 2289724788}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6802', 'nonce': 2289724788}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6804', 'nonce': 2289724788}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6803', 'nonce': 2289724788}]}, 'public_addr': '172.21.15.38:6801/2289724788', 'cluster_addr': '172.21.15.38:6802/2289724788', 'heartbeat_back_addr': '172.21.15.38:6804/2289724788', 'heartbeat_front_addr': '172.21.15.38:6803/2289724788', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46b46f54-04d0-46c0-809c-64fa385e6780', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 21, 'down_at': 39, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 3833397294}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6809', 'nonce': 3833397294}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6811', 'nonce': 3833397294}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 3833397294}]}, 'public_addr': '172.21.15.64:6808/3833397294', 'cluster_addr': '172.21.15.64:6809/3833397294', 'heartbeat_back_addr': '172.21.15.64:6811/3833397294', 'heartbeat_front_addr': '172.21.15.64:6810/3833397294', 'state': ['exists']}, {'osd': 5, 'uuid': 'f7a50ac0-eba9-4602-b90a-2dbc4bb20593', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 36, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 3915827095}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6801', 'nonce': 3915827095}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6803', 'nonce': 3915827095}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 3915827095}]}, 'public_addr': '172.21.15.64:6800/3915827095', 'cluster_addr': '172.21.15.64:6801/3915827095', 'heartbeat_back_addr': '172.21.15.64:6803/3915827095', 'heartbeat_front_addr': '172.21.15.64:6802/3915827095', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '081024c0-2482-4846-a7bd-594a58f2ceef', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 36, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1144336651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6805', 'nonce': 1144336651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6807', 'nonce': 1144336651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1144336651}]}, 'public_addr': '172.21.15.64:6804/1144336651', 'cluster_addr': '172.21.15.64:6805/1144336651', 'heartbeat_back_addr': '172.21.15.64:6807/1144336651', 'heartbeat_front_addr': '172.21.15.64:6806/1144336651', 'state': ['exists']}, {'osd': 7, 'uuid': '6234c757-8f7b-497e-9347-4a7983df592c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 370645108}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6813', 'nonce': 370645108}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6815', 'nonce': 370645108}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 370645108}]}, 'public_addr': '172.21.15.64:6812/370645108', 'cluster_addr': '172.21.15.64:6813/370645108', 'heartbeat_back_addr': '172.21.15.64:6815/370645108', 'heartbeat_front_addr': '172.21.15.64:6814/370645108', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:56.081253+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:55.642638+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:55.131979+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:55.147738+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.907705+0000', 'dead_epoch': 38}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.533669+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.310102+0000', 'dead_epoch': 36}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.959957+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.0', 'osds': [3, 5]}, {'pgid': '2.1', 'osds': [7, 5]}, {'pgid': '2.4', 'osds': [2, 5]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-03T05:27:41.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:27:41.648 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:41.647+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:41.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:41.747 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:41.747+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:41.825 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:41.825 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:41.836 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:41.849 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:41.848+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:41.949 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:41.949+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:41.969 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:41.970 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:41.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:42.050 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:42.050+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:42.151 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:42.151 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:42.153 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:42.153+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:42.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:42.254 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:42.253+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:42.339 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:42.339 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:42.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:42.396 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:42.395+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:42.478 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:42.477+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:42.530 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:42.530 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:42.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:42.546 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:27:42.546 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:27:42.592 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:42.592+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:42.641 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:27:42.642 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:27:42.658 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:42.658+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:42.696 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 210.873193095, 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:42.697 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:42.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 210.61436129200001, 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:42.701 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:42.702 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:42.702 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:42.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:42.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:42.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:42.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:42.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:42.705 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.542449337000001, 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:42.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.709 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.452306085000004, 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:42.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:42.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:42.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:27:42.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:27:42.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:42.713 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:42.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:42.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:42.713 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:42.713 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:42.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:42.745 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:27:42.745 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:27:42.795 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:42.795+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:42.841 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:27:42.841 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:27:42.860 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:42.860+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:42.884 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:42.884 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:42.884 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:42.884 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:42.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:42.960 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:42.960+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:43.002 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:27:43.004 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:27:43.062 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:43.062+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:43.073 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:43.073 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:43.073 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:43.073 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:43.073 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:43.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:43.121 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:27:43.121 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:27:43.169 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:43.169+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:43.225 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:27:43.226 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":46,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:27:43.241 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-06-03T05:27:43.241 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-03T05:27:43.241 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-06-03T05:27:43.264 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:43.264+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:43.275 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:43.275 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:43.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:43.275 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.313218163999998, 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:43.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:27:43.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:27:43.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:43.280 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.077848054, 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:43.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:27:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:27:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.285 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:43.285 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:43.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:43.287 INFO:tasks.ceph.mon.a:Started 2024-06-03T05:27:43.287 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-06-03T05:27:43.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:27:43.320 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:27:43.320 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:27:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:27:43.326 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:27:43.365 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:43.365+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:43.465 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:43.466+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:43.489 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:43.490 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:43.490 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:43.490 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:43.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:43.568 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:43.567+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:43.667 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:43.667+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:43.672 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:43.673 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:43.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:43.770 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:43.770+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:43.869 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:43.869+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:43.901 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:43.901 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:43.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:43.901 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:43.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.901 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:27:43.902 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:43.902 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 206.23423218600001, 2024-06-03T05:27:43.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:27:43.902 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:43.902 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:43.902 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:43.902 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:27:43.902 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:27:43.903 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:43.903 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:27:43.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.903 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:43.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:43.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:43.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:43.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:43.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:43.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:43.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:43.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:27:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:27:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:27:43.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:27:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:27:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:27:43.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:27:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:27:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:27:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:27:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:27:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:27:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:27:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.086258446000002, 2024-06-03T05:27:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:27:43.911 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:27:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:27:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:43.912 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:27:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:43.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:43.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:43.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:27:43.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:27:43.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:43.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:27:43.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:27:43.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:43.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:27:43.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:27:43.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:43.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:27:43.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:27:43.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:43.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:27:43.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:27:43.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:43.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:43.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:43.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:27:43.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:27:43.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.920 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:43.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:43.920 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:43.920 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:43.922 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:43.936 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:27:43.937 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:27:43.950 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-06-03T05:27:43.018580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":12,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":192692,"kb_used_data":3544,"kb_used_omap":10,"kb_used_meta":189045,"kb_avail":660410188,"statfs":{"total":676457349120,"available":676260032512,"internally_reserved":0,"allocated":3629056,"data_stored":1964651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":193582187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.682484"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-06-03T05:27:04.213675+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:27:04.213675+0000","last_peered":"2024-06-03T05:27:04.213675+0000","last_clean":"2024-06-03T05:27:04.213675+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:27:04.213675+0000","last_undegraded":"2024-06-03T05:27:04.213675+0000","last_fullsized":"2024-06-03T05:27:04.213675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":28,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-06-03T05:27:04.145730+0000","last_change":"2024-06-03T05:26:04.857151+0000","last_active":"2024-06-03T05:27:04.145730+0000","last_peered":"2024-06-03T05:27:04.145730+0000","last_clean":"2024-06-03T05:24:13.082657+0000","last_became_active":"2024-06-03T05:26:04.857151+0000","last_became_peered":"2024-06-03T05:26:04.857151+0000","last_unstale":"2024-06-03T05:27:04.145730+0000","last_undegraded":"2024-06-03T05:27:04.145730+0000","last_fullsized":"2024-06-03T05:26:04.787809+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:06:27.875765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":69,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-06-03T05:27:04.206056+0000","last_change":"2024-06-03T05:26:04.893483+0000","last_active":"2024-06-03T05:27:04.206056+0000","last_peered":"2024-06-03T05:27:04.206056+0000","last_clean":"2024-06-03T05:25:47.358980+0000","last_became_active":"2024-06-03T05:26:04.893483+0000","last_became_peered":"2024-06-03T05:26:04.893483+0000","last_unstale":"2024-06-03T05:27:04.206056+0000","last_undegraded":"2024-06-03T05:27:04.206056+0000","last_fullsized":"2024-06-03T05:26:04.881186+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:16:46.977266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":50,"state":"active+clean+remapped","last_fresh":"2024-06-03T05:27:04.901368+0000","last_change":"2024-06-03T05:26:56.490616+0000","last_active":"2024-06-03T05:27:04.901368+0000","last_peered":"2024-06-03T05:27:04.901368+0000","last_clean":"2024-06-03T05:27:04.901368+0000","last_became_active":"2024-06-03T05:26:56.459009+0000","last_became_peered":"2024-06-03T05:26:56.459009+0000","last_unstale":"2024-06-03T05:27:04.901368+0000","last_undegraded":"2024-06-03T05:27:04.901368+0000","last_fullsized":"2024-06-03T05:27:04.901368+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:12:14.369529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":68,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-06-03T05:27:04.197094+0000","last_change":"2024-06-03T05:25:47.369969+0000","last_active":"2024-06-03T05:27:04.197094+0000","last_peered":"2024-06-03T05:27:04.197094+0000","last_clean":"2024-06-03T05:25:31.097366+0000","last_became_active":"2024-06-03T05:25:47.369969+0000","last_became_peered":"2024-06-03T05:25:47.369969+0000","last_unstale":"2024-06-03T05:27:04.197094+0000","last_undegraded":"2024-06-03T05:27:04.197094+0000","last_fullsized":"2024-06-03T05:25:47.343518+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:03:21.221713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":71,"reported_epoch":50,"state":"active+clean+remapped","last_fresh":"2024-06-03T05:27:04.902977+0000","last_change":"2024-06-03T05:26:55.359495+0000","last_active":"2024-06-03T05:27:04.902977+0000","last_peered":"2024-06-03T05:27:04.902977+0000","last_clean":"2024-06-03T05:27:04.902977+0000","last_became_active":"2024-06-03T05:26:55.349708+0000","last_became_peered":"2024-06-03T05:26:55.349708+0000","last_unstale":"2024-06-03T05:27:04.902977+0000","last_undegraded":"2024-06-03T05:27:04.902977+0000","last_fullsized":"2024-06-03T05:27:04.902977+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T06:15:29.680294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":50,"state":"active+clean+remapped","last_fresh":"2024-06-03T05:27:04.188426+0000","last_change":"2024-06-03T05:26:55.393944+0000","last_active":"2024-06-03T05:27:04.188426+0000","last_peered":"2024-06-03T05:27:04.188426+0000","last_clean":"2024-06-03T05:27:04.188426+0000","last_became_active":"2024-06-03T05:26:55.377925+0000","last_became_peered":"2024-06-03T05:26:55.377925+0000","last_unstale":"2024-06-03T05:27:04.188426+0000","last_undegraded":"2024-06-03T05:27:04.188426+0000","last_fullsized":"2024-06-03T05:27:04.188426+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:01:40.201295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-06-03T05:27:04.201688+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:27:04.201688+0000","last_peered":"2024-06-03T05:27:04.201688+0000","last_clean":"2024-06-03T05:27:04.201688+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:27:04.201688+0000","last_undegraded":"2024-06-03T05:27:04.201688+0000","last_fullsized":"2024-06-03T05:27:04.201688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":87,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-06-03T05:27:04.902109+0000","last_change":"2024-06-03T05:26:56.251989+0000","last_active":"2024-06-03T05:27:04.902109+0000","last_peered":"2024-06-03T05:27:04.902109+0000","last_clean":"2024-06-03T05:27:04.902109+0000","last_became_active":"2024-06-03T05:26:55.711401+0000","last_became_peered":"2024-06-03T05:26:55.711401+0000","last_unstale":"2024-06-03T05:27:04.902109+0000","last_undegraded":"2024-06-03T05:27:04.902109+0000","last_fullsized":"2024-06-03T05:27:04.902109+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:12:18.487760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378675,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":183815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411376,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":183815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":18,"seq":77309411359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":319488,"data_stored":131441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":843776,"data_stored":581655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411379,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27848,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343992,"statfs":{"total":96636764160,"available":96608247808,"internally_reserved":0,"allocated":843776,"data_stored":581655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476784,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":183815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":118455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":55834574896,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:27:43.951 INFO:tasks.thrashosds.thrasher:Setting 2.6 to [0, 5, 2, 7] 2024-06-03T05:27:43.951 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.6', '0', '5', '2', '7'] 2024-06-03T05:27:43.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.6 0 5 2 7 2024-06-03T05:27:43.991 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:43.991+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:44.025 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:19.853 53808== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:44.056 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:27:44.057 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:44.067 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:44.070 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:44.070+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:44.081 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:19.909 53808== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:44.098 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":46,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:27:44.113 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:19.936 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:44.171 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:44.171+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:44.175 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:19.998 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:44.202 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:44.202 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:44.202 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:27:44.202 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:27:44.202 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:44.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:44.272 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:44.272+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:44.284 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:20.112 53808== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:44.370 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:20.193 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:44.372 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:44.372+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:44.443 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:27:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:27:44.443 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 225.57311365999999, 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:27:44.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 224.398157328, 2024-06-03T05:27:44.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.454 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 224.39719489399999, 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:27:44.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.459 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 224.396404891, 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.460 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:27:44.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:27:44.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 224.36175102999999, 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:44.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 224.361061533, 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:27:44.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:44.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 224.36038818399999, 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:27:44.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.481 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 224.28795506899999, 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:27:44.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:27:44.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:27:44.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 223.52996787699999, 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:27:44.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:27:44.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 223.281317556, 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.497 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 223.28007216899999, 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:27:44.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 223.27901266000001, 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:27:44.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:27:44.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:27:44.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:27:44.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:27:44.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 223.246164265, 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.513 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:27:44.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:27:44.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:27:44.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:27:44.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:27:44.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:27:44.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.518 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.519 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.519 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:44.519 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 223.24507913400001, 2024-06-03T05:27:44.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:27:44.519 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.519 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.520 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:27:44.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.520 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:44.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:44.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:27:44.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:27:44.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:27:44.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:27:44.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:44.527 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 223.24406211600001, 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.528 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:27:44.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:27:44.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 223.224278599, 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:44.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:27:44.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 219.09967060599999, 2024-06-03T05:27:44.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:27:44.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:27:44.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 219.09885094200001, 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:27:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:27:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:27:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:27:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 219.09814999299999, 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:27:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:27:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.69109626100001, 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:27:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:44.559 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:44.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:44.561 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:44.561+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:44.661 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:44.661+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:44.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:27:44.713+0000 7f29faffd640 0 --2- 172.21.15.38:0/2714593014 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29fc0aa720 0x7f29fc138bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-03T05:27:44.749 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:44.749 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:44.749 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:27:44.749 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:44.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:44.762 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:44.762+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:44.879 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:44.879+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:44.893 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:27:44.893 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:27:44.894 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:44.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:44.988 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:44.988+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:45.024 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:27:45.065 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:45.065+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:45.083 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:45.083 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:27:45.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:27:45.083 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:27:45.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.084 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:27:45.084 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:45.084 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 206.211367268, 2024-06-03T05:27:45.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:27:45.085 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:45.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:45.085 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:45.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:45.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:27:45.086 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:45.086 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:27:45.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.086 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:45.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:45.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:45.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:45.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:45.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:45.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:45.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:45.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:27:45.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:27:45.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:45.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:27:45.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:27:45.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:45.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:27:45.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:27:45.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:45.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:27:45.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:27:45.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:45.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:27:45.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:27:45.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:45.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:27:45.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:27:45.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:45.095 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:45.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:45.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.096 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:27:45.096 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:45.097 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 86.093408122, 2024-06-03T05:27:45.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:27:45.097 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:45.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:45.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:45.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:45.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:27:45.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:45.098 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:27:45.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:45.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:45.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:45.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:45.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:45.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:45.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:45.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:45.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:27:45.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:27:45.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:45.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:27:45.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:27:45.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:45.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:27:45.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:27:45.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:45.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:27:45.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:27:45.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:45.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:27:45.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:27:45.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:45.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:45.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:45.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:27:45.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:27:45.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:45.106 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:45.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:45.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:45.106 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:45.107 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:45.108 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:45.166 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:45.166+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:45.245 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:45.245 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:45.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:45.296 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:45.296+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:45.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:27:45.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-06-03T05:27:45.373+0000 7f99367fc640 0 --2- 172.21.15.38:0/918767920 >> [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] conn(0x7f99380b10b0 0x7f993813cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-03T05:27:45.373 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:45.374+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:45.434 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:45.435 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:45.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:45.469 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:45.469+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:45.571 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:45.570+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:45.617 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:45.617 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:45.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:45.617 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:27:45.617 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:45.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:45.672 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:45.672+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:45.773 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:45.773+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:45.803 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:45.803 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:45.803 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:45.803 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:45.817 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:45.854 INFO:teuthology.orchestra.run.smithi038.stderr:set 2.6 pg_upmap_items mapping to [0->5,2->7] 2024-06-03T05:27:45.874 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:45.874+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:45.967 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:45.967 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:45.968 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:45.968 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:45.968 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:45.975 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:45.976+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:45.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:46.021 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:27:46.021 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:27:46.076 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:46.076+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:46.158 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:27:46.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:27:46.160 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:46.160 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:46.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:46.177 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:46.177+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:46.261 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:27:46.261 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:27:46.290 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:46.290+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:46.325 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:46.325 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:46.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:46.377 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:27:46.378 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:27:46.379 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:46.379+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:27:46.478 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:27:46.478 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:27:46.480 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:46.479+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:46.480 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:27:46.509 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:46.509 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:46.518 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:46.580 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:46.580+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:46.589 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:27:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:27:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:27:46.595 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:27:46.681 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:46.681+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:46.694 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:46.694 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:46.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:46.782 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:46.782+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:46.879 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:46.879 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:46.883 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:46.883+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:46.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:46.936 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:27:46.993 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:46.992+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:47.070 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:47.070 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:47.079 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:47.086 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:47.086+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:47.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:27:47.189 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:47.188+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:47.267 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:47.267 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:47.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:47.321 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:47.320+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:47.389 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:47.388+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:47.413 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:47.413 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:47.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:47.490 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:47.489+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:47.554 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:47.555 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:47.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:47.607 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:47.607+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:47.692 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:47.692+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:47.696 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:47.696 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:47.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:47.793 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:47.792+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:47.829 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":46,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:27:47.879 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:47.880 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:47.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:47.894 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:47.893+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:48.016 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:48.016+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:48.029 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:48.029 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:48.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:48.096 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:48.095+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:48.196 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:48.196+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:48.212 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:48.212 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:48.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:48.298 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:48.297+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:48.359 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:48.359 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:48.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:48.411 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:48.409+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:48.500 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:48.499+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:48.541 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:48.541 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:48.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:48.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:27:48.601 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:48.601+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:48.701 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:48.701+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:48.727 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:48.727 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:48.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:48.802 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:48.802+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:48.902 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:27:48.902 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:27:48.903 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:48.903+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:48.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:27:48.960 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:48.960 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:48.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:49.013 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:49.011+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:49.027 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:27:49.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:27:49.105 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:49.105+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:49.131 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:27:49.131 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:27:49.150 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:49.150 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:49.150 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:49.151 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:49.151 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:49.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:49.227 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:27:49.228 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:27:49.228 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:49.227+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:49.332 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:49.331+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:49.366 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 217.545754872, 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:49.367 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:27:49.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:49.371 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 217.28692306900001, 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:27:49.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.375 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 53.215011113999999, 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:49.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:27:49.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 53.124867862000002, 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:49.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:49.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:49.383 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:49.385 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:49.428 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:49.426+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:49.440 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:27:49.535 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:49.535+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:49.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:27:49.536 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:27:49.537 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:27:49.563 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:49.563 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:49.563 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:49.563 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:49.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:49.630 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:27:49.631 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:27:49.631 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:49.631+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:27:49.634 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:27:49.711 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:49.710+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:49.753 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:49.753 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:49.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:49.847 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:49.846+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:49.913 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:49.913+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:49.942 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:49.943 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:49.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:49.943 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:49.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.943 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:49.943 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:49.943 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 53.981423868999997, 2024-06-03T05:27:49.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:27:49.943 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:27:49.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:49.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.947 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 53.746053758999999, 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:49.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:27:49.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:49.952 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:49.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:50.013 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:50.014+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:50.115 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:50.115+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:50.127 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:50.127 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:50.127 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:50.127 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:50.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:50.215 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:50.215+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:50.270 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:50.270 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:50.270 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:50.270 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:50.270 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:50.279 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:50.316 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:50.316+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:50.417 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:50.416+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:50.460 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:50.460 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:50.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:50.460 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:50.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.460 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:27:50.460 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:50.460 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 212.79334374300001, 2024-06-03T05:27:50.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:27:50.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:27:50.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 92.645370002999996, 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:50.465 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:27:50.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:50.469 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:50.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:50.493 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:27:50.519 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:50.520+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:50.614 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:27:50.615 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:50.618 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:50.618+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:50.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:50.719 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:50.718+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:50.796 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:50.797 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:50.797 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:27:50.797 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:27:50.797 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:50.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:50.820 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:50.819+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:50.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:27:50.875 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [6, 4] live_osds: [0, 1, 2, 3, 5, 7] 2024-06-03T05:27:50.876 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:27:50.876 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-03T05:27:50.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-06-03T05:27:50.924 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:50.924+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:50.990 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:27:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 232.12069388399999, 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:27:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:27:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 230.945737552, 2024-06-03T05:27:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:27:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:27:51.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:27:51.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:27:51.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:27:51.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:27:51.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:27:51.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.003 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.003 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:51.004 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 230.944775118, 2024-06-03T05:27:51.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:27:51.004 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:27:51.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:51.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:51.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:27:51.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:27:51.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:27:51.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:27:51.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:27:51.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:27:51.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:27:51.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:27:51.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:27:51.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:27:51.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 230.943985115, 2024-06-03T05:27:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:27:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:27:51.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:27:51.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:27:51.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:27:51.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:27:51.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:27:51.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:27:51.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:27:51.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:27:51.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.019 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.020 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:51.020 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 230.90933125399999, 2024-06-03T05:27:51.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:27:51.020 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:27:51.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:51.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:51.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:27:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:27:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:27:51.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 230.908641757, 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.027 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 230.90796840799999, 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:51.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:51.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:27:51.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 230.83553529299999, 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:27:51.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 230.07754810099999, 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:27:51.044 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:27:51.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:27:51.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.82889778000001, 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.053 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.053 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.053 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.053 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:27:51.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.054 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:51.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:51.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:27:51.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:27:51.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:27:51.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:27:51.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:27:51.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:27:51.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:27:51.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:27:51.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:27:51.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:27:51.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:27:51.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.060 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.061 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.061 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:51.061 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.82765239299999, 2024-06-03T05:27:51.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:27:51.061 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.061 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:27:51.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.82659288400001, 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.067 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:27:51.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.073 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.073 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:51.073 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.79374448900001, 2024-06-03T05:27:51.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:27:51.073 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.074 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:27:51.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.074 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:27:51.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:27:51.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:51.079 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.79265935800001, 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:51.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:51.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:27:51.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:27:51.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:27:51.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:27:51.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:27:51.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.086 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.086 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.086 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:51.086 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.79164234000001, 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:51.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:27:51.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:27:51.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:27:51.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:27:51.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:27:51.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:27:51.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:27:51.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:27:51.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.093 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:51.093 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:51.093 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.771858823, 2024-06-03T05:27:51.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:27:51.093 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.094 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.094 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.094 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:27:51.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.094 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:51.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:51.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:27:51.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:27:51.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:27:51.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:27:51.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:27:51.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.099 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 225.64725082999999, 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:27:51.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:27:51.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.105 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 225.64643116600001, 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:27:51.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:51.111 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 225.64573021699999, 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:27:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:27:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:27:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:27:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:27:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 219.23867648500001, 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:27:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:27:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:27:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:27:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.122 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:51.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:51.124 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:51.123+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:51.198 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:07.840 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:51.225 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:51.225+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:51.298 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:27:51.299 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:51.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:51.326 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:51.325+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:51.414 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:08.056 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:51.427 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:51.426+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:51.480 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:51.481 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:51.481 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:27:51.481 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:27:51.481 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:51.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:51.528 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:51.527+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:51.562 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":46,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 54, 2024-06-03T05:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "ebc09f7e-f094-4421-826f-479fc526fc62", 2024-06-03T05:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-06-03T05:23:10.198378+0000", 2024-06-03T05:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-06-03T05:27:50.388146+0000", 2024-06-03T05:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-06-03T05:26:03.427745+0000", 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-06-03T05:26:52.558950+0000", 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5806080, 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "noscrub", 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 6, 2024-06-03T05:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 2, 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "jewel", 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-06-03T05:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-06-03T05:23:56.753149+0000", 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 1, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-06-03T05:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "16", 2024-06-03T05:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-06-03T05:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-06-03T05:27:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-06-03T05:27:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 8, 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 8, 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.25, 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2, 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2, 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-06-03T05:27:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "rbd", 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-06-03T05:24:12.898324+0000", 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 1, 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-06-03T05:27:51.610 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 8, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 8, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 8, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 8, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 8, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-06-03T05:27:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "24", 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-06-03T05:27:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-06-03T05:27:51.613 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-06-03T05:27:51.614 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 5, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 5, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.5, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.5, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.5, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-06-03T05:27:51.615 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "32b256d9-eee7-435b-8aec-fef2ba90366c", 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:51.616 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 16, 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 39, 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.617 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-06-03T05:27:51.618 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 905141127 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.619 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/905141127", 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6811/905141127", 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/905141127", 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6813/905141127", 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "371c2e00-54db-430b-8497-a4c9ff6753b9", 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:51.620 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0.9170379638671875, 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 44, 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:27:51.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:51.622 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3490615740 2024-06-03T05:27:51.623 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6810/3490615740", 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6812/3490615740", 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6815/3490615740", 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6814/3490615740", 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-06-03T05:27:51.624 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "e8145335-ec5d-4dd1-9250-a4ab5b159d29", 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 46, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.625 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.627 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3359326199 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/3359326199", 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/3359326199", 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/3359326199", 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/3359326199", 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c", 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 19, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 45, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:51.629 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:27:51.630 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2289724788 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/2289724788", 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/2289724788", 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/2289724788", 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/2289724788", 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:51.632 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "46b46f54-04d0-46c0-809c-64fa385e6780", 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 0, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 39, 2024-06-03T05:27:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6808", 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.634 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6809", 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6811", 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.635 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6810", 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3833397294 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6808/3833397294", 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6809/3833397294", 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6811/3833397294", 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6810/3833397294", 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: "exists" 2024-06-03T05:27:51.636 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f7a50ac0-eba9-4602-b90a-2dbc4bb20593", 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 36, 2024-06-03T05:27:51.637 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6800", 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.638 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6801", 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6803", 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6802", 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3915827095 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6800/3915827095", 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6801/3915827095", 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6803/3915827095", 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6802/3915827095", 2024-06-03T05:27:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "081024c0-2482-4846-a7bd-594a58f2ceef", 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 0, 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 0, 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 36, 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6804", 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:51.642 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6805", 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6807", 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:27:51.643 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6806", 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1144336651 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6804/1144336651", 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6805/1144336651", 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1144336651", 2024-06-03T05:27:51.644 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6806/1144336651", 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "exists" 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "6234c757-8f7b-497e-9347-4a7983df592c", 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-06-03T05:27:51.645 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 45, 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6812", 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-06-03T05:27:51.646 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6813", 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6815", 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:27:51.647 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.64:6814", 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 370645108 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.64:6812/370645108", 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.64:6813/370645108", 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.64:6815/370645108", 2024-06-03T05:27:51.648 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.64:6814/370645108", 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:51.649 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:56.081253+0000", 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:55.642638+0000", 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:55.131979+0000", 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:51.651 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:55.147738+0000", 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.907705+0000", 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 38 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.533669+0000", 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:51.653 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.310102+0000", 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 36 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-06-03T05:27:51.654 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-06-03T05:23:54.959957+0000", 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [], 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [ 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "2.0", 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: 5 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "2.1", 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: 5 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "2.4", 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: 2, 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: 5 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.656 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": {}, 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-06-03T05:27:51.657 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:51.658 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:51.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:27:51.660 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 212.757998302, 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:27:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.640039156, 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:27:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:27:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:27:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:27:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:51.671 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:51.671 INFO:tasks.thrashosds.thrasher:j is {'epoch': 54, 'fsid': 'ebc09f7e-f094-4421-826f-479fc526fc62', 'created': '2024-06-03T05:23:10.198378+0000', 'modified': '2024-06-03T05:27:50.388146+0000', 'last_up_change': '2024-06-03T05:26:03.427745+0000', 'last_in_change': '2024-06-03T05:26:52.558950+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-03T05:23:56.753149+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-06-03T05:24:12.898324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5, 'score_stable': 5, 'optimal_score': 0.5, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '32b256d9-eee7-435b-8aec-fef2ba90366c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 16, 'up_thru': 39, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6809', 'nonce': 905141127}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6811', 'nonce': 905141127}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6816', 'nonce': 905141127}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6813', 'nonce': 905141127}]}, 'public_addr': '172.21.15.38:6809/905141127', 'cluster_addr': '172.21.15.38:6811/905141127', 'heartbeat_back_addr': '172.21.15.38:6816/905141127', 'heartbeat_front_addr': '172.21.15.38:6813/905141127', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '371c2e00-54db-430b-8497-a4c9ff6753b9', 'up': 1, 'in': 1, 'weight': 0.9170379638671875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 44, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6810', 'nonce': 3490615740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6812', 'nonce': 3490615740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6815', 'nonce': 3490615740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6814', 'nonce': 3490615740}]}, 'public_addr': '172.21.15.38:6810/3490615740', 'cluster_addr': '172.21.15.38:6812/3490615740', 'heartbeat_back_addr': '172.21.15.38:6815/3490615740', 'heartbeat_front_addr': '172.21.15.38:6814/3490615740', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e8145335-ec5d-4dd1-9250-a4ab5b159d29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6805', 'nonce': 3359326199}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6806', 'nonce': 3359326199}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6808', 'nonce': 3359326199}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6807', 'nonce': 3359326199}]}, 'public_addr': '172.21.15.38:6805/3359326199', 'cluster_addr': '172.21.15.38:6806/3359326199', 'heartbeat_back_addr': '172.21.15.38:6808/3359326199', 'heartbeat_front_addr': '172.21.15.38:6807/3359326199', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6801', 'nonce': 2289724788}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6802', 'nonce': 2289724788}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6804', 'nonce': 2289724788}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6803', 'nonce': 2289724788}]}, 'public_addr': '172.21.15.38:6801/2289724788', 'cluster_addr': '172.21.15.38:6802/2289724788', 'heartbeat_back_addr': '172.21.15.38:6804/2289724788', 'heartbeat_front_addr': '172.21.15.38:6803/2289724788', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46b46f54-04d0-46c0-809c-64fa385e6780', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 21, 'down_at': 39, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 3833397294}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6809', 'nonce': 3833397294}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6811', 'nonce': 3833397294}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 3833397294}]}, 'public_addr': '172.21.15.64:6808/3833397294', 'cluster_addr': '172.21.15.64:6809/3833397294', 'heartbeat_back_addr': '172.21.15.64:6811/3833397294', 'heartbeat_front_addr': '172.21.15.64:6810/3833397294', 'state': ['exists']}, {'osd': 5, 'uuid': 'f7a50ac0-eba9-4602-b90a-2dbc4bb20593', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 36, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 3915827095}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6801', 'nonce': 3915827095}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6803', 'nonce': 3915827095}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 3915827095}]}, 'public_addr': '172.21.15.64:6800/3915827095', 'cluster_addr': '172.21.15.64:6801/3915827095', 'heartbeat_back_addr': '172.21.15.64:6803/3915827095', 'heartbeat_front_addr': '172.21.15.64:6802/3915827095', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '081024c0-2482-4846-a7bd-594a58f2ceef', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 36, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1144336651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6805', 'nonce': 1144336651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6807', 'nonce': 1144336651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1144336651}]}, 'public_addr': '172.21.15.64:6804/1144336651', 'cluster_addr': '172.21.15.64:6805/1144336651', 'heartbeat_back_addr': '172.21.15.64:6807/1144336651', 'heartbeat_front_addr': '172.21.15.64:6806/1144336651', 'state': ['exists']}, {'osd': 7, 'uuid': '6234c757-8f7b-497e-9347-4a7983df592c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 370645108}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6813', 'nonce': 370645108}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6815', 'nonce': 370645108}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 370645108}]}, 'public_addr': '172.21.15.64:6812/370645108', 'cluster_addr': '172.21.15.64:6813/370645108', 'heartbeat_back_addr': '172.21.15.64:6815/370645108', 'heartbeat_front_addr': '172.21.15.64:6814/370645108', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:56.081253+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:55.642638+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:55.131979+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:55.147738+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.907705+0000', 'dead_epoch': 38}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.533669+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.310102+0000', 'dead_epoch': 36}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-03T05:23:54.959957+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.0', 'osds': [3, 5]}, {'pgid': '2.1', 'osds': [7, 5]}, {'pgid': '2.4', 'osds': [2, 5]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-03T05:27:51.671 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-06-03T05:27:51.672 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:51.659+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:51.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:51.761 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:51.760+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:51.838 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:51.838 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:51.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:51.889 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:51.888+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:51.959 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:27:51.959 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:27:51.963 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:51.962+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:52.038 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:52.038 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:52.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:52.070 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:27:52.071 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:27:52.071 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:52.070+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:52.165 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:52.165+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:52.167 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:27:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:27:52.205 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:52.206 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:52.215 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:52.266 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:52.266+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:52.271 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:27:52.272 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:27:52.276 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:27:52.364 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:52.364 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:52.366 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:52.367+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:52.369 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:27:52.369 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:27:52.372 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:27:52.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:52.467 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:52.467+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:52.482 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:27:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:27:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:27:52.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:27:52.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:27:52.520 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:52.520 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:52.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:52.572 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:52.570+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:52.669 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:52.668+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:52.712 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:52.712 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:52.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:52.806 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:52.805+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:52.871 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:52.870+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:52.915 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:52.915 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:52.924 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:53.009 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:53.008+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:53.073 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:53.073+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:53.118 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:53.119 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:53.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:53.174 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:53.174+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:53.275 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:53.275+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:53.304 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:53.304 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:53.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:53.376 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:53.376+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:53.453 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:53.454 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:53.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:53.507 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:53.506+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:53.577 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:53.576+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:53.596 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:53.596 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:53.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:53.678 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:53.678+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:53.741 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:27:53.779 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:53.779+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:53.792 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:53.792 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:53.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:53.879 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:53.880+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:53.947 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:53.948 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:53.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:53.980 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:53.980+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:54.082 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:54.081+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:54.136 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:54.136 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:54.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:54.182 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:54.181+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:54.279 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:54.279 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:54.283 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:54.283+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:54.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:54.384 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:54.384+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:54.465 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:54.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:54.484 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:54.483+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:54.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:27:54.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:27:54.620 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:54.619+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:54.648 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:54.648 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:54.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:54.686 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:54.685+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:54.750 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:11.393 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:54.751 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:27:54.751 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:27:54.787 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:54.787+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:54.793 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:27:54.793 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:27:54.803 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:54.871 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:27:54.871 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:27:54.873 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:11.515 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:54.889 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:54.888+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:54.976 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:54.976 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:54.985 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:27:54.985 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:27:54.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:55.029 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:55.027+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:55.081 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:27:55.081 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:27:55.090 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:55.090+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:55.136 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:11.778 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:27:55.174 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:55.174 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:55.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:55.208 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:27:55.209 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:55.208+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:55.292 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:55.293+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:55.319 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:27:55.320 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:27:55.322 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:27:55.357 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:55.358 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:55.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:55.358 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:27:55.358 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:55.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:55.394 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:55.394+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:55.424 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:27:55.425 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:27:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:27:55.430 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:27:55.495 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:55.495+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:55.529 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:55.529 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:55.529 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:55.529 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:55.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:55.595 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:55.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:55.675 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:55.675 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:55.675 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:55.676 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:55.676 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:55.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:55.697 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:55.696+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:55.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:27:55.803 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:55.802+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:55.825 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:27:55.835 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:55.835 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:55.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:55.835 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 224.01236925000001, 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:27:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:55.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 223.75353744700001, 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:55.840 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:55.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:27:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:27:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:55.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:27:55.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 59.681625492000002, 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:55.845 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:55.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:55.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:27:55.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:55.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:27:55.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:27:55.850 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:55.850 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:55.850 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:55.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 59.591482239999998, 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:27:55.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:55.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:27:55.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:27:55.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:27:55.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:55.855 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:55.856 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-06-03T05:27:55.856 INFO:tasks.mon_recovery:removing mon b 2024-06-03T05:27:55.857 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-03T05:27:55.857 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:27:55.858 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:55.874 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:27:55.863+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 53806) UID: 0 2024-06-03T05:27:55.875 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:27:55.865+0000 9f9e640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-06-03T05:27:55.898 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:55.898+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:55.999 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:55.998+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:56.074 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:56.074 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:56.074 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:56.074 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:56.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:56.127 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:56.125+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:56.201 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:56.201+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:56.261 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:56.261 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:56.261 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:56.261 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:56.261 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:56.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:56.302 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:56.302+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:56.374 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:32.201 53808== 2024-06-03T05:27:56.374 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:32.201 53808== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:27:56.385 INFO:tasks.ceph.mon.b.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:27:56.403 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:56.402+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:56.448 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:56.448 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:56.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:56.448 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:56.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.448 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:56.448 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:56.448 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.486704107000001, 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:56.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.452 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.251333997000003, 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:27:56.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:56.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:27:56.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:27:56.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:27:56.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.457 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:56.458 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:56.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:56.543 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:56.542+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:56.628 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:56.625+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:56.643 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:56.644 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:56.653 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:27:56.673 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [6, 4] live_osds: [0, 1, 2, 3, 5, 7] 2024-06-03T05:27:56.673 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:27:56.705 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:56.705+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:56.806 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:56.806+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:56.827 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:56.828 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:56.836 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:27:56.907 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:56.907+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:56.974 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:56.974 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:56.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:56.974 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:56.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.974 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 219.307740854, 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:27:56.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:56.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:27:56.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 99.159767114000005, 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:27:56.979 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:56.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:56.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:27:56.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:27:56.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:56.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:27:56.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.983 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:56.984 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:56.984 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:56.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:27:57.028 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:57.027+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:57.111 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:57.111+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:57.126 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:57.126 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:57.126 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:27:57.126 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:57.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:27:57.220 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:27:57.220+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:27:57.310 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:27:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:27:57.311 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:57.311 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:57.311+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:57.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:27:57.412 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:57.412+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:57.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:27:57.514 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:57.514+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:57.546 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:27:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:27:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:27:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 238.67479627399999, 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:27:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 237.49983994199999, 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:27:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:27:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:27:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 237.49887750799999, 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:27:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:27:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:27:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:27:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:27:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 237.498087505, 2024-06-03T05:27:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:27:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:27:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:27:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:27:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:27:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 237.46343364399999, 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:27:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:27:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 237.462744147, 2024-06-03T05:27:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:27:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 237.46207079800001, 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:27:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:27:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:27:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 237.38963768299999, 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:27:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:27:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 236.63165049099999, 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:27:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:27:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:27:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 236.38300017, 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:27:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:27:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:27:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:27:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:27:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 236.38175478299999, 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:27:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:27:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:27:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:27:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:27:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:27:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:27:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:27:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:27:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:27:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:27:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:27:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:27:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:27:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 236.380695274, 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:27:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:27:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:27:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 236.347846879, 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:27:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:27:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 236.34676174800001, 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:27:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:27:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:27:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:27:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:27:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 236.34574473000001, 2024-06-03T05:27:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:27:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:27:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:27:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:27:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:27:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:27:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:27:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:27:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:27:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:27:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:27:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:27:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:27:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:27:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:27:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:27:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:27:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:27:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 236.325961213, 2024-06-03T05:27:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:27:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:27:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:27:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 232.20135321999999, 2024-06-03T05:27:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:27:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:27:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:27:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:27:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:27:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:27:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 232.20053355600001, 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:27:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 232.19983260699999, 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:27:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 225.79277887500001, 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:27:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:27:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:27:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:27:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:57.672 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:57.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:27:57.674 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:57.673+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:57.775 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:57.774+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:57.807 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:57.807 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:57.807 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:27:57.807 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:57.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:27:57.875 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:57.875+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:57.950 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:57.950 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:27:57.951 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:27:57.951 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:27:57.951 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:57.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:27:58.003 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:58.003+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:58.078 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:58.078+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:58.098 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:27:58.098 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:27:58.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:27:58.098 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:27:58.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.098 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:27:58.098 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:58.098 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 219.22667619399999, 2024-06-03T05:27:58.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:58.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:58.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.102 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.108717048000003, 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:27:58.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:58.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:27:58.107 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:27:58.107 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:27:58.108 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:27:58.179 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:58.178+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:58.244 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:58.244 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:58.245 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:58.245 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:58.254 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:27:58.297 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:58.295+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:58.381 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:58.381+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:58.430 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:58.430 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:58.430 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:58.430 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:58.430 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:58.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:27:58.482 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:58.482+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:58.582 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:58.582+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:58.614 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:58.614 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:58.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:58.614 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:27:58.614 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:58.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:27:58.708 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:58.707+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:58.784 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:58.784+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:58.797 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:58.798 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:58.798 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:58.798 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:58.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:27:58.893 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:58.892+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:58.982 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:58.982 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:58.982 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:58.983 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:58.983 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:58.986 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:58.985+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:58.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:27:59.087 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:59.086+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:59.172 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:59.172 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:59.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:59.172 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.351645596, 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:59.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:59.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:27:59.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:27:59.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:59.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:27:59.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:27:59.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:59.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:27:59.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:27:59.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:59.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.178 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.092813793, 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:27:59.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:27:59.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:59.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:27:59.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:27:59.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 63.020901838, 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:59.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:59.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:59.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:59.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:27:59.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:59.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:27:59.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:27:59.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:59.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:27:59.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:27:59.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:59.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:27:59.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:27:59.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:59.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:27:59.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:27:59.189 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.190 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:59.190 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.190 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:27:59.190 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:59.190 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 62.930758586000003, 2024-06-03T05:27:59.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:27:59.190 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:59.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:59.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:27:59.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:27:59.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.194 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:59.195 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.195 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.195 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:59.195 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:59.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:27:59.197 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:59.196+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:59.297 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:27:59.296+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:27:59.376 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:59.376 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:59.376 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:59.376 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:59.385 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:27:59.428 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:59.426+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:59.499 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:27:59.498+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:27:59.560 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:27:59.560 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:27:59.560 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:59.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:27:59.600 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:27:59.600+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:27:59.701 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:59.700+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:59.750 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:59.750 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:27:59.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:27:59.750 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:27:59.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.750 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:59.750 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:59.750 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 63.788380232999998, 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:27:59.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:59.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 63.553010123, 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:59.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:27:59.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:27:59.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.759 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:59.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:27:59.759 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:27:59.759 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:59.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:27:59.802 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:27:59.801+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:27:59.903 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:27:59.902+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:27:59.936 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:27:59.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:27:59.936 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:27:59.936 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:27:59.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:00.030 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:00.029+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:00.105 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:00.104+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:00.118 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:00.119 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:00.119 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:00.119 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:00.119 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:00.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:00.213 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:00.212+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:00.306 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 222.640227307, 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 102.49225356700001, 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:00.315 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:00.316 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:00.306+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:00.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:00.407 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:00.408+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:00.463 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:00.463 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:00.463 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:00.463 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:00.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:00.515 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:00.514+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:00.609 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:00.609+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:00.648 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:00.648 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:00.648 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:00.648 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:00.648 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:00.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:00.710 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:00.710+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:00.841 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:00.841 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:00.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:00.841 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:00.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.841 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:00.842 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:00.842 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 241.971294847, 2024-06-03T05:28:00.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:00.842 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.842 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.842 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:00.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 240.796338515, 2024-06-03T05:28:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.848 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:00.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:00.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:00.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.854 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 240.795376081, 2024-06-03T05:28:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:00.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:00.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:00.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.864 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 240.79458607800001, 2024-06-03T05:28:00.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:00.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:00.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 240.759932217, 2024-06-03T05:28:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:00.874 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.875 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:00.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:00.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:00.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:00.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 240.75924272, 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:00.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:00.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.888 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 240.75856937099999, 2024-06-03T05:28:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:00.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 240.686136256, 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:00.895 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:00.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 239.928149064, 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.901 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:00.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 239.67949874300001, 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:00.906 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:00.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:00.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.911 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 239.678253356, 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:00.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 239.67719384700001, 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:00.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 239.64434545200001, 2024-06-03T05:28:00.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:00.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 239.64326032100001, 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.928 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:00.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 239.64224330299999, 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.933 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:00.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:00.938 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 239.62245978600001, 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:00.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:00.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:00.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 235.497851793, 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:00.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 235.49703212899999, 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:00.949 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:00.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.954 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 235.49633118, 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:00.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:00.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:00.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.08927744799999, 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:00.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:00.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:00.964 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:00.964 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:00.841+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:00.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:01.007 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:01.007+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:01.067 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:01.067+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:01.139 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:01.140 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:01.140 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:01.140 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:01.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:01.169 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:01.168+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:01.270 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:01.270+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:01.322 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:01.322 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:01.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:01.371 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:01.370+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:01.472 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:01.471+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:01.508 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:01.508 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:01.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:01.508 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:01.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.508 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:01.508 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:01.508 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 222.63678946799999, 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:01.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 102.518830322, 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:01.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:01.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:01.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:01.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:01.518 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:01.518 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:01.519 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:01.573 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:01.572+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:01.666 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:01.667 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:01.667 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:01.667 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:01.674 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [6, 4] live_osds: [0, 1, 2, 3, 5, 7] 2024-06-03T05:28:01.674 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:28:01.674 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-06-03T05:28:01.674 INFO:tasks.ceph.osd.6:Restarting daemon 2024-06-03T05:28:01.674 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-06-03T05:28:01.676 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:01.676 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:01.676+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:01.678 INFO:tasks.ceph.osd.6:Started 2024-06-03T05:28:01.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:01.773 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-06-03T05:28:01.774 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:01.774+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:28:01.776 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-03T05:28:01.851 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:01.851 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:01.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:01.875 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:01.875+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:28:01.958 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-06-03T05:28:01.959 INFO:tasks.ceph.mon.b:Stopped 2024-06-03T05:28:01.959 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-06-03T05:28:01.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:28:01.975 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-06-03T05:28:02.003 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:02.002+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:02.035 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:02.035 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:02.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:02.035 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:02.035 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:02.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:02.089 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:02.088+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:02.179 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:02.178+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:02.221 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:02.221 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:02.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:02.279 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:02.279+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:02.380 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:02.380+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:02.405 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:02.405 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:02.405 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:02.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:02.498 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:02.497+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:02.582 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:02.581+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:02.603 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:02.603 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 230.782481295, 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:02.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:02.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:02.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:02.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 230.523649492, 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:02.608 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:02.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:02.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:02.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.451737537, 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:02.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:02.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:02.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:02.616 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.361594284999995, 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:02.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:02.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:02.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:02.620 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:02.620 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:02.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:02.683 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:02.683+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:02.777 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:28:02.784 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:02.784+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:02.797 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:02.797 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:02.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:02.885 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:02.885+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:02.981 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:02.981 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:02.981 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:02.986 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:02.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:02.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:03.087 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:03.086+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:03.129 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:03.129 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:03.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:03.129 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:03.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.129 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.167990463999999, 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:28:03.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:28:03.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.134 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:03.134 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:03.134 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.932620353999994, 2024-06-03T05:28:03.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:28:03.134 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:03.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:03.134 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:03.134 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:28:03.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:03.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:28:03.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:28:03.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:03.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:28:03.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:28:03.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.142 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:03.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.143 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:03.143 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:03.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:03.187 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:03.188+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:03.304 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:03.303+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:03.319 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:03.319 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:03.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:03.390 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:03.390+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:03.491 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:03.491+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:03.505 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:03.505 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:03.505 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:03.506 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:03.506 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:03.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:03.599 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:03.599+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:03.692 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 226.02558739099999, 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 105.877613651, 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:03.701 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:03.701 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:03.693+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:03.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:03.793 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:03.794+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:03.847 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:03.847 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:03.847 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:03.847 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:03.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:03.903 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:03.902+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:03.993 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:03.993 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:03.993 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:03.993 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:03.993 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:03.995 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:03.996+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:04.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:04.097 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:04.097+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:04.198 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:04.198+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:04.228 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 245.35935807000001, 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 244.18440173799999, 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 244.18343930399999, 2024-06-03T05:28:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 244.182649301, 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 244.14799543999999, 2024-06-03T05:28:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 244.14730594299999, 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 244.14663259400001, 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:04.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:04.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.270 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 244.07419947899999, 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:04.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.272 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:04.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:04.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:04.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:04.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:04.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:04.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.277 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 243.31621228700001, 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:04.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:04.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 243.067561966, 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:04.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:04.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 243.06631657899999, 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:04.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:04.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:04.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:04.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:04.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.295 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 243.06525707, 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:04.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:04.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:04.302 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 243.032408675, 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:04.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:04.308 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 243.031323544, 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:04.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:04.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:04.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 243.030306526, 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:04.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:04.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:04.321 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 243.010523009, 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:04.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:04.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:04.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:04.327 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 238.88591501600001, 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:04.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 238.88509535200001, 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:04.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:04.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 238.88439440299999, 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:04.339 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 232.47734067100001, 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:04.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.348 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.349 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.349 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:04.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:04.351 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:04.350+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:04.451 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:04.451+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:04.548 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:04.548 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:04.548 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:04.552 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:04.552+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:04.558 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:04.653 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:04.653+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:04.745 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:04.745 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:04.746 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:04.746 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:04.753 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:04.753+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:04.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:04.854 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:04.854+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:04.933 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:04.933 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:04.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:04.933 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:04.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:04.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:04.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 226.06163426399999, 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:04.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:04.937 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 105.943675118, 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:04.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:04.941 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:04.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:04.989 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:04.988+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:05.055 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:05.055+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:05.092 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:05.092 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:05.092 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:05.092 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:05.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:05.186 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:05.186+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:05.257 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:05.257+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:05.275 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:05.276 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:05.276 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:05.276 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:05.276 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:05.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:05.371 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:05.371+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:05.458 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:05.459 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:05.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:05.459 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:05.459 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:05.459 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:05.459+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:05.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:05.560 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:05.560+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:05.641 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:05.641 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:05.641 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:05.641 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:05.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:05.694 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:05.693+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:05.762 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:05.762+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:05.791 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:28:05.825 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:05.825 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:05.825 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:05.826 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:05.826 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:05.834 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:05.862 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:05.862+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:05.963 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:05.963+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:06.021 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 234.20091820799999, 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:06.022 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:06.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 233.942086405, 2024-06-03T05:28:06.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:06.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:06.030 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.870174449999993, 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:06.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.780031198000003, 2024-06-03T05:28:06.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:06.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:06.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:06.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:06.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.038 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:06.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.038 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:06.038 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:06.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:06.083 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:06.083+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:06.167 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:06.167+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:06.173 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:06.173 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:06.173 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:06.173 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:06.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:06.267 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:06.268+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:06.360 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:06.361 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:06.361 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:06.361 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:06.361 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:06.369 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:06.369+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:06.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:06.470 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:06.470+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:06.553 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:06.553 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:06.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:06.553 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:06.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.553 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:06.553 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:06.553 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.59101038, 2024-06-03T05:28:06.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:28:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:06.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.355640269999995, 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:06.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:06.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:06.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:06.561 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:06.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:06.605 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:06.604+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:06.671 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:06.672+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:06.736 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:06.737 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:06.737 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:06.737 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:06.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:06.772 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:06.772+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:06.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:06.873 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-06-03T05:28:06.874 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:06.874+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:28:06.876 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-03T05:28:06.922 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:06.923 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:06.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:06.974 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:06.974+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:07.091 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:07.091+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:07.109 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:07.109 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:07.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:07.109 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:07.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.109 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:07.109 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:07.109 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 229.442695665, 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:07.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.113 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 109.294721925, 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:07.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:07.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:07.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:07.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:07.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:07.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:07.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:07.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:07.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:07.118 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:07.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:07.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:07.118 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:07.118 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:07.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:07.175 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:07.175+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:07.252 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:07.252 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:07.252 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:07.252 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:07.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:07.305 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:07.305+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:07.379 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:07.379+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:07.394 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:07.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:07.394 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:07.394 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:07.394 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:07.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:07.478 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:07.478+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:07.579 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:07.579+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:07.626 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:07.626 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:07.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:07.626 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:07.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 248.75709522400001, 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.627 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:07.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:07.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:07.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 247.58213889199999, 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:07.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:07.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:07.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 247.58117645799999, 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:07.637 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.642 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 247.580386455, 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:07.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 247.54573259399999, 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.648 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:07.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:07.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:07.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 247.54504309699999, 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.655 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:07.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:07.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:07.660 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 247.54436974800001, 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:07.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 247.47193663300001, 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:07.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:07.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 246.71394944100001, 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.671 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:07.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:07.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 246.46529912, 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.676 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:07.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:07.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 246.46405373299999, 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.681 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:07.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:07.686 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 246.462994224, 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:07.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.691 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 246.430145829, 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:07.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:07.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 246.429060698, 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:07.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.701 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 246.42804368, 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.702 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:07.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:07.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 246.40826016299999, 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.707 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:07.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:07.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 242.28365217000001, 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.712 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:07.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 242.28283250600001, 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.717 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:07.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.722 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 242.28213155700001, 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:07.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 235.87507782500001, 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:07.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:07.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:07.731 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:07.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:07.733 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:07.733+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:07.834 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:07.835+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:07.907 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:07.907 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:07.908 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:07.908 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:07.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:07.960 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:07.960+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:08.036 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:08.036+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:08.062 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:08.062 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:08.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:08.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-06-03T05:28:08.155 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:08.155+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:08.238 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:08.238+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:08.249 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.37856895799999, 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:08.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:08.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.260609812, 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:08.259 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:08.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:08.339 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:08.339+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:08.408 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:08.408 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:08.408 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:08.408 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:08.419 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:08.465 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:08.465+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:08.542 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:08.542+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:08.556 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:08.556 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:08.556 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:08.557 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:08.557 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:08.565 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:08.650 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:08.650+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:08.742 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:08.743 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:08.743 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:08.743+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:08.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:08.843 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:08.844+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:08.925 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:08.925 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:08.925 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:08.925 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:08.934 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:08.975 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:08.975+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:09.045 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:09.046+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:09.105 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:09.106 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:09.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:09.147 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:09.147+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:09.247 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:09.248+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:09.299 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.477494888, 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.218663085, 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.146751129999998, 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.056607877999994, 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:09.316 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:09.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:09.359 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:09.359+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:09.449 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:09.449+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:09.496 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:09.496 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:09.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:09.590 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:09.590+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:09.651 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:09.651+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:09.680 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:09.680 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:09.680 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:09.681 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:09.681 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:09.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:09.774 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:09.774+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:09.852 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:09.853+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:09.870 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:09.870 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.909074308000001, 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:09.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:28:09.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:09.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.673704197999996, 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:09.875 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:28:09.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:09.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:09.879 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:09.880 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:09.954 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:09.954+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:10.055 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:10.055+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:10.057 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:10.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:10.057 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:10.057 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:10.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:10.155 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:10.156+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:10.241 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:10.241 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:10.250 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:10.292 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:10.292+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:10.358 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:10.358+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:10.428 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:10.428 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:10.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:10.429 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:10.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.429 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:10.429 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:10.429 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 232.76157385400001, 2024-06-03T05:28:10.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:10.429 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:10.429 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:10.429 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:10.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:10.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:10.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:10.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 112.61360011399999, 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:10.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:10.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:10.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:10.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:10.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:10.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:10.438 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:10.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:10.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:10.438 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:10.438 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:10.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:10.484 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:10.484+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:10.560 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:10.560+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:10.586 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:10.587 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:10.587 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:10.587 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:10.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:10.681 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:10.681+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:10.762 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:10.762+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:10.770 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:10.770 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:10.770 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:10.770 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:10.770 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:10.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:10.863 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:10.863+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:10.964 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:10.965+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:11.014 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 252.13395140599999, 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 250.958995074, 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 250.95803264, 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 250.95724263700001, 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 250.922588776, 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 250.921899279, 2024-06-03T05:28:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 250.92122592999999, 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 250.848792815, 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 250.09080562299999, 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.84215530200001, 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.840909915, 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.83985040600001, 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.80700201100001, 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.80591688000001, 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.090 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:11.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:11.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:11.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:11.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:11.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:11.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:11.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:11.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:11.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:11.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:11.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:11.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:11.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:11.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:11.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:11.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:11.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:11.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:11.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.099 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.099 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.099 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.099 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:11.100 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.80489986200001, 2024-06-03T05:28:11.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:11.100 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.100 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:11.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:11.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.106 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:11.106 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:11.106 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.78511634500001, 2024-06-03T05:28:11.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:11.106 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.107 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:11.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.107 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:11.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:11.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:11.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:11.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:11.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:11.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:11.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:11.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:11.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:11.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:11.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:11.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:11.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:11.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:11.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:11.115 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.115 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.116 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:11.116 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:11.116 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 245.66050835199999, 2024-06-03T05:28:11.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:11.116 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.116 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.117 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.117 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:11.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.117 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:11.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:11.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:11.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:11.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:11.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:11.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:11.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:11.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:11.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:11.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:11.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:11.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:11.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:11.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:11.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:11.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:11.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:11.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:11.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:11.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.124 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.125 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:11.125 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:11.125 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 245.65968868799999, 2024-06-03T05:28:11.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:11.125 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.126 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:11.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.126 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:11.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:11.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:11.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:11.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:11.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:11.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:11.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:11.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:11.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:11.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:11.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:11.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:11.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 245.658987739, 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:11.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:11.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:11.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:11.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 239.25193400699999, 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:11.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.144 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.144 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:11.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:11.146 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:11.146+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:11.247 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:11.247+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:11.270 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:28:11.280 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:11.280 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:11.280 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:11.280 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:11.286 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-06-03T05:28:11.286 INFO:tasks.mon_recovery:causing some monitor log activity 2024-06-03T05:28:11.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '1 of 30' 2024-06-03T05:28:11.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:11.374 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:11.374+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:11.449 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:11.449+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:11.462 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:11.462 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:11.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:11.550 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:11.550+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:11.563 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:11.563 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:11.607 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 232.73646580799999, 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 112.618506662, 2024-06-03T05:28:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:11.616 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:11.617 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:11.662 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:11.662+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:11.665 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:11.666 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:11.752 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:11.753+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:11.780 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:11.781 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:11.796 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:11.797 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:11.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:11.854 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:11.853+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:11.877 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:11.900 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:11.900 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:11.926 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:11.954 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:11.955+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:11.992 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-06-03T05:28:11.994 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:28:11.995 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-03T05:28:11.995 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:11.995 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:12.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:12.024 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:12.025 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:12.027 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:12.055 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:12.055+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:12.128 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:12.129 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:12.133 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:12.158 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:12.155+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:12.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:12.257 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:12.256+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:12.331 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:12.332 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:12.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:12.357 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:12.356+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:12.469 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:12.469+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:12.484 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:12.484 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:12.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:12.559 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:12.558+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:12.660 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:12.659+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:12.677 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:12.678 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:12.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:12.761 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:12.761+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:12.765 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:11.025 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:12.825 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:11.085 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:12.826 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:12.826 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:12.836 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:12.861 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:12.861+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:12.963 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:12.962+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:12.982 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:28:13.018 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:13.018 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:13.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:13.064 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:13.063+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:13.171 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:13.171+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:13.212 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:13.212 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:13.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:13.271 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:13.271+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:13.367 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:13.367+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:13.399 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:13.392+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:28:13.407 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:13.407 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:13.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:13.468 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:13.467+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:13.584 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:13.583+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:13.597 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:13.597 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:13.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:13.669 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:13.668+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:13.770 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:13.769+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:13.790 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:13.790 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:13.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:13.833 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:12.093 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:13.872 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:13.871+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:13.938 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:13.938 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:13.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:13.973 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:13.973+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:14.073 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:14.073+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:14.122 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:14.122 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:14.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:14.134 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:28:14.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '2 of 30' 2024-06-03T05:28:14.207 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:12.467 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:14.217 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:14.212+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:14.265 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:14.265 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:14.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:14.275 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:14.274+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:14.327 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:12.587 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:14.410 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:14.410 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:14.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:14.427 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:14.428 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:14.428 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:14.428+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:14.477 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:14.477+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:14.520 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:14.521 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:14.557 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:14.557 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:14.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:14.614 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:14.614 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:14.615 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:14.614+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:14.624 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:12.884 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:14.678 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:14.678+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:14.698 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:14.698 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:14.707 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:14.711 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:14.712 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:14.715 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:14.779 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:14.779+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:14.826 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:14.827 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:14.828 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:14.842 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:14.842 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:14.842 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:14.842 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:14.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:14.893 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:14.893+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:14.917 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:13.178 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:14.921 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:14.921 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:14.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:14.981 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:14.981+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:14.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:28:15.030 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:15.030 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:15.030 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:15.030 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:15.030 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:15.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:15.121 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:15.121+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:15.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '3 of 30' 2024-06-03T05:28:15.182 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:15.181+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:15.215 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:13.475 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:15.226 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:15.226 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:15.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:15.321 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:15.321+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:15.384 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:15.383+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:15.430 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:15.430 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:15.430 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:15.430 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:15.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:15.484 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:15.484+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:15.504 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:13.765 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:15.586 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:15.586+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:15.620 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:15.620 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:15.620 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:15.620 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:15.620 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:15.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:15.687 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:15.686+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:15.788 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:15.787+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:15.802 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:14.063 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:15.811 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 243.989400711, 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 243.73056890800001, 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 79.658656953000005, 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 79.568513701000001, 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:15.827 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:15.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:15.889 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:15.889+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:15.971 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:15.971 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:15.971 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:15.971 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:15.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:15.989 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:15.989+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:16.090 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:16.090+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:16.097 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:14.357 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:16.156 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:16.157 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:16.157 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:16.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:16.207 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:16.207+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:16.281 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:28:16.291 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:16.291+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:16.351 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 80.388655069999999, 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:28:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:28:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 80.153284959999993, 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:28:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:28:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:16.360 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:16.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:16.392 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:16.391+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:16.493 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:16.493+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:16.499 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:16.499 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:16.499 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:16.500 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:16.509 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:16.594 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:16.594+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:16.631 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:14.892 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:16.683 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:16.684 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:16.688 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:14.948 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:16.693 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:16.695 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:16.694+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:16.796 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:16.795+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:16.830 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:16.830 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:16.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:16.830 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 239.16382075300001, 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:16.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:16.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 119.015847013, 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:16.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:16.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:16.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:16.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:16.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:16.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:16.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.839 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:16.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:16.839 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:16.839 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:16.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:16.896 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:16.895+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:16.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:28:16.974 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:16.974 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:16.974 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:16.974 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:16.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:16.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:16.997 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:16.997+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:17.113 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:17.114+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:17.161 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:17.162 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:17.162 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:17.162 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:17.162 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:17.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:17.196 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:17.196 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:17.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '4 of 30' 2024-06-03T05:28:17.239 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:17.239+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:17.301 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:17.301+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:17.324 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:17.324 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:17.355 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 258.48592527800002, 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 257.310968946, 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 257.31000651199997, 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 257.30921650900001, 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:17.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:17.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 257.27456264800003, 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.378 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:17.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:17.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 257.27387315099998, 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:17.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 257.27319980200002, 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:17.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:17.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 257.200766687, 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:17.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:17.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 256.44277949500002, 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:17.399 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 256.19412917400001, 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:17.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:17.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 256.19288378700003, 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.410 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:17.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 256.19182427800001, 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:17.415 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:17.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:17.420 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 256.15897588299998, 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 256.15789075200001, 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.426 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:17.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:17.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 256.15687373399999, 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:17.431 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 256.13709021699998, 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:17.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:17.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 252.012482224, 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:17.441 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:17.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 252.01166255999999, 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:17.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.451 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 252.010961611, 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:17.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 245.60390787899999, 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:17.457 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:17.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:17.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:17.461 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:17.461 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:17.461 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:17.462 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:17.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:17.463 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:17.462+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:17.521 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:17.524 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:17.563 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:17.562+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:17.600 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:17.600 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:17.600 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:17.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:17.625 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:17.625 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:17.627 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:17.663 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:17.664+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:17.728 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:17.728 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:17.732 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:17.732 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:17.751 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore bluefs device info [] 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-06-03T05:28:17.752 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:28:17.754 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:28:17.754 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-03T05:28:17.774 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:17.774+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:17.788 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:17.788 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:17.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:17.882 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:17.881+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:17.967 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:17.967+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:17.988 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:17.988 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:17.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:18.068 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:18.068+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:18.135 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:18.136 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:18.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:18.147 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '5 of 30' 2024-06-03T05:28:18.168 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:18.168+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:18.270 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:18.270+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:18.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:28:18.332 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:18.332 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:18.344 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:18.370 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:18.369+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:18.470 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:18.471+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:18.532 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:18.532 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:18.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:18.584 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:18.583+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:18.672 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:18.672+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:18.729 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:18.729 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:18.739 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:18.780 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:18.780+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:18.827 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:35.469 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:18.833 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:35.475 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:18.861 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:35.503 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:18.873 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:18.873+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:18.904 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:54.727 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:18.907 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:54.730 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:18.923 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:18.924 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:18.934 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:18.937 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:54.759 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:18.980 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:18.980+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:19.079 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:19.079+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:19.085 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:19.085 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:19.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:19.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '6 of 30' 2024-06-03T05:28:19.176 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:19.176+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:19.277 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:19.276+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:19.278 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:19.278 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:19.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:19.377 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:19.377+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:19.425 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:19.425 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:19.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:19.478 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:19.479+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:19.578 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:19.578+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:19.610 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:19.611 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:19.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:19.679 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:19.678+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:19.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:28:19.800 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:19.800+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:19.817 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:19.818 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:19.829 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:19.873 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:28:19.880 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:19.879+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:19.982 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:19.981+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:20.031 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:20.031 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:20.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:20.083 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:20.082+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:20.085 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:20.085 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:20.180 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:20.180 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:20.184 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:20.184+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:20.186 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:20.186 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:20.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:20.206 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '7 of 30' 2024-06-03T05:28:20.291 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:20.291+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:20.294 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:20.294 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:20.336 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:20.336 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:20.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:20.386 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:20.386+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:20.429 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:20.429 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:20.432 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:20.487 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:20.487+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:20.546 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:20.546 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:20.550 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:20.550 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:20.551 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:20.557 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:20.588 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:20.588+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:20.657 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:20.657 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:20.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:20.703 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:20.703+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:20.741 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 261.87202617299999, 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.742 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:20.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:20.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:20.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:20.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:20.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:20.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:20.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:20.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:20.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:20.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:20.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 260.69706984099997, 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.749 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:20.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 260.696107407, 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.755 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:20.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:20.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 260.69531740399998, 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.760 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 260.660663543, 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 260.659974046, 2024-06-03T05:28:20.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:20.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 260.65930069699999, 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.776 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:20.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 260.58686758200002, 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 259.82888038999999, 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:20.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:20.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 259.58023006899998, 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.792 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:20.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 259.578984682, 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.797 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 259.57792517299998, 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 259.54507677800001, 2024-06-03T05:28:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.812 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 259.54399164699998, 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:20.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:20.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 259.54297462900001, 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:20.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:20.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 259.52319111200001, 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:20.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:20.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.39858311899999, 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:20.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.39776345499999, 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:20.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:20.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:20.838 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.397062506, 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:20.843 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 248.99000877399999, 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:20.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:20.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:20.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:20.847 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:20.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:20.849 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:20.849+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:20.950 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:20.949+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:20.983 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:20.983 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:20.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:21.077 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:21.076+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:21.167 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:21.167+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:21.184 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:21.184 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:21.184 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:21.185 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:21.185 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:21.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:21.278 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:21.278+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:21.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '8 of 30' 2024-06-03T05:28:21.363 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:21.362+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:21.373 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:21.373 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:21.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:21.373 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:21.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.374 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:21.374 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:21.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 242.50220290799999, 2024-06-03T05:28:21.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:21.374 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:21.374 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:21.374 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:21.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:21.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:21.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.384243762, 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:21.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:21.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:21.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:21.382 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:21.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:21.455 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:21.455+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:21.544 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:21.544 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:21.545 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:21.545 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:21.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:21.596 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:21.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:21.661 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:21.661+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:21.730 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:21.730 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:21.739 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:21.782 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:21.781+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:21.859 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:21.859+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:21.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:28:21.918 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:21.918 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:21.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:21.918 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:21.918 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:21.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:21.971 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:21.970+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:22.029 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:20.289 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:22.061 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:22.061+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:22.119 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:22.119 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:22.120 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:22.120 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:22.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:22.170 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:22.170+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:22.263 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:22.263+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:22.301 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:22.302 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:22.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:22.363 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:22.363+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:22.464 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:22.464+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:22.475 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 250.65496951899999, 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 250.396137716, 2024-06-03T05:28:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.324225760999994, 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.234082509000004, 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:22.493 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:22.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:22.579 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:22.579+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:22.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:28:22.668 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:22.669 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:22.669 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:22.669+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:22.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:22.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:22.766 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:22.767+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:22.867 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:22.867+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:22.874 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:22.874 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:22.874 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:22.874 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:22.883 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:22.931 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:22.932 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:22.968 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:22.969+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:23.039 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:23.039 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:23.069 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:23.069+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 87.124600006999998, 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:28:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:28:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.889229897000007, 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:23.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:23.094 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:23.095 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:23.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:23.145 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:23.146 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:23.170 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:23.171+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:23.242 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:23.242 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:23.244 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:23.271 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:23.271+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:23.280 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:23.281 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:23.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:23.338 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:23.338 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:23.340 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:23.372 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:23.372+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:23.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '9 of 30' 2024-06-03T05:28:23.439 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:23.439 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:23.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:23.470 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:28:23.473 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:23.473+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:23.484 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:23.484 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:23.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:23.579 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:23.579+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:23.674 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:23.674 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:23.675 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:23.675+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:23.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:23.776 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:23.776+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:23.817 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:23.817 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:23.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:23.877 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:23.877+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:23.963 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:23.963 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:23.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:24.017 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:24.017+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:24.079 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:24.079+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:24.106 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:24.106 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:24.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:24.200 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:24.200+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:24.281 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:24.281+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:24.291 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:24.291 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:24.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:24.382 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:24.382+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:24.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '10 of 30' 2024-06-03T05:28:24.476 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:24.476 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:24.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:24.510 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:24.510+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:24.583 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:24.584+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:24.619 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:24.619 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:24.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:24.684 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:24.685+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:24.765 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:24.766 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:24.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:24.785 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:24.785+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:24.810 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:28:24.810 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:28:24.810 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:28:24.810 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:28:24.810 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:28:24.810 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:28:24.810 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:28:24.810 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-06-03T05:28:24.811 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-06-03T05:28:24.811 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:28:24.811 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:28:24.811 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:28:24.812 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:28:24.813 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-03T05:28:24.895 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:24.896+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:24.909 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:24.910 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:24.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:24.987 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:24.988+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:25.089 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:25.089+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:25.096 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:25.097 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:25.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:25.191 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:25.191+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:25.282 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:25.282 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:25.291 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:25.292 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:25.291+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:25.391 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:25.391+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:25.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '11 of 30' 2024-06-03T05:28:25.444 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:28:25.484 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:25.484 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:25.492 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:25.492+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:25.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:28:25.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:25.593 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:25.593+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:25.694 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:25.694+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:25.695 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:25.695 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:25.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:25.795 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:25.796+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:25.879 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:25.879 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:25.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:25.930 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:25.930+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:25.998 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:25.998+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:26.019 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:26.020 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:26.094 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:26.094 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:26.099 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:26.099+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:26.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:26.133 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:26.134 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:26.200 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:26.201+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:26.249 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:26.306 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 90.345343778, 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:28:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 90.109973667999995, 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:28:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:26.316 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:26.316 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:26.307+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:26.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:26.363 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:26.364 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:26.373 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:26.401 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:26.402+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:26.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '12 of 30' 2024-06-03T05:28:26.474 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:26.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:26.501 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:26.502 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:26.502+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:26.503 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:26.503 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:26.514 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:26.588 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:28:26.595 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:26.595 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:26.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:26.603 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:26.603+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:26.704 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:26.705+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:26.710 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:26.710 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:26.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:26.806 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:26.806+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:26.899 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 249.23281508900001, 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 129.08484134899999, 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:26.908 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:26.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:26.910 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:26.910+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:27.011 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:27.011+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:27.042 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:27.043 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:27.043 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:27.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:27.112 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:27.112+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:27.213 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:27.213+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:27.225 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:27.225 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:27.225 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:27.225 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:27.225 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:27.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:27.292 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:25.552 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:27.296 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:25.556 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:27.314 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:27.315+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:27.415 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:27.415+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:27.421 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:27.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 268.551604586, 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.422 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:27.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:27.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 267.37664825399997, 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.428 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 267.37568582, 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 267.37489581699998, 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 267.340241956, 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:27.444 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:27.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 267.339552459, 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:27.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:27.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:27.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 267.33887910999999, 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:27.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:27.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 267.26644599500003, 2024-06-03T05:28:27.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:27.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.467 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 266.508458803, 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:27.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:27.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 266.25980848199998, 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:27.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 266.258563095, 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.478 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:27.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:27.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 266.25750358599998, 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:27.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 266.22465519100001, 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.489 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:27.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:27.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 266.22357005999999, 2024-06-03T05:28:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:27.494 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:27.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 266.22255304200002, 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:27.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:27.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:27.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:27.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 266.20276952500001, 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:27.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 262.07816153200002, 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:27.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:27.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:27.518 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 262.07734186800002, 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:27.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:27.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.523 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 262.076640919, 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:27.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:27.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.66958718699999, 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:27.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.532 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:27.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '13 of 30' 2024-06-03T05:28:27.535 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:27.534+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:27.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:27.635 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:27.635+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:27.668 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:27.668 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:27.669 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:27.669 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:27.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:27.735 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:27.736+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:27.813 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:27.813 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:27.813 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:27.813 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:27.813 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:27.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:27.867 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:27.867+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:27.938 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:27.938+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:27.960 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:27.960 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 249.08890434099999, 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:27.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:27.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 128.97094519500001, 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:27.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:27.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:27.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:27.969 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:27.969 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:27.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:28.039 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:28.039+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:28.121 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:28.121 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:28.122 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:28.122 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:28.131 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:28.172 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:28.172+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:28.241 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:28.241+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:28.306 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:28.306 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:28.306 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:28.306 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:28.306 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:28.316 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:28.358 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:28.358+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:28.442 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:28.443+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:28.490 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:28.491 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:28.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:28.491 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:28.491 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:28.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:28.544 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:28.544+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:28.604 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:28:28.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:28:28.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '14 of 30' 2024-06-03T05:28:28.650 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:28.650 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:28.651 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:28.651 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:28.651 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:28.651+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:28.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:28.746 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:28.746+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:28.847 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:28.848+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:28.860 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:28.860 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:28.861 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:28.861 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:28.861 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:28.870 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:28.916 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:28.916 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:28.949 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:28.948+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:29.010 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:29.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:29.049 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:29.050+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:29.062 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:29.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:29.142 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:29.143 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:29.150 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:29.151+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:29.259 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:29.260 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:29.260 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:29.257+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:29.261 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:29.298 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:29.298 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:29.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:29.352 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:29.352+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:29.360 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:29.360 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:29.362 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:29.453 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:29.453+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:29.476 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:29.476 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:29.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:29.485 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:29.485 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:29.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:29.553 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:29.553+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:29.568 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:28:29.655 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:29.655+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:29.701 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:29.701 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:29.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:29.755 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:29.755+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:29.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:29.856 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:29.856+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:29.899 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:29.899 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:29.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:29.993 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:29.993+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:30.057 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:30.057+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:30.084 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:30.084 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:30.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:30.178 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:30.178+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:30.258 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:30.259+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:30.268 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:30.268 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:30.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:30.360 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:30.360+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:30.412 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:30.412 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:30.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:30.461 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:30.461+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:30.553 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:30.554 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:30.562 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:30.562+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:30.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:30.682 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:30.682+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:30.696 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:30.696 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:30.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:30.777 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:30.777+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:30.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '15 of 30' 2024-06-03T05:28:30.885 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:30.885+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:30.891 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:30.891 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:30.902 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:30.964 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:30.964+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:31.038 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:31.038 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:31.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation histogram block [] [] 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator score block 2024-06-03T05:28:31.054 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-06-03T05:28:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:28:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:28:31.056 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-03T05:28:31.065 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:31.065+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:31.166 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:31.166+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:31.183 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:31.183 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:31.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:31.233 INFO:tasks.ceph.osd.6.smithi064.stderr:==00:00:00:29.493 55327== Warning: unimplemented fcntl command: 1038 2024-06-03T05:28:31.266 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:31.266+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:31.331 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:31.331 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:31.341 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:31.367 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:31.367+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:31.470 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:31.471+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:31.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:28:31.518 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:31.519 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:31.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:31.574 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:31.574+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:31.586 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:28:31.668 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:31.669+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:31.769 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:31.769 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:31.770 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:31.770+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:31.770 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:31.770 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:31.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:31.770 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:31.770 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:31.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:31.864 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:31.864 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:31.870 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:31.870+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:31.957 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:31.957 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:31.957 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:31.957 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:31.967 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:31.998 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:31.999 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:32.001 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:32.000+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:32.071 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:32.072+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:32.100 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:32.101 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:32.104 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:32.148 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:32.149 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:32.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:32.167 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '16 of 30' 2024-06-03T05:28:32.172 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:32.172+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:32.200 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:32.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:32.213 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:32.273 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:32.273+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:32.326 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:32.326 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:32.333 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:32.352 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:32.352 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 260.53054421000002, 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:32.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:32.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 260.271712407, 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:32.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:32.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.361 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:32.362 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:32.362 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 96.199800452000005, 2024-06-03T05:28:32.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:32.362 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:32.362 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:32.362 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:32.362 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:32.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:32.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 96.109657200000001, 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:32.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:32.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:32.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:32.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:32.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:32.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:32.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:32.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:32.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:32.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.370 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:32.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.371 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:32.371 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:32.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:32.373 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:32.374+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:32.474 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:32.475+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:32.549 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:32.549 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:32.549 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:32.549 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:32.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:32.601 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:32.601+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:32.676 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:32.677+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:32.738 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:32.738 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:32.739 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:32.739 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:32.739 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:32.748 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:32.790 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:32.790+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:32.878 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:32.879+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 96.973452785999996, 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:32.935 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:28:32.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:32.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:32.939 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 96.738082676000005, 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:32.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:28:32.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:32.944 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:32.944 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:32.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:32.986 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:32.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:33.081 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:33.081+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:33.120 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:33.120 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:33.120 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:33.121 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:33.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:33.215 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:33.215+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:33.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '17 of 30' 2024-06-03T05:28:33.293 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:33.293+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:33.315 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:33.315 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:33.315 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:33.315 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:33.315 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:33.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:33.387 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:28:33.388 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:33.388+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:33.484 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:33.484+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:33.515 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:33.515 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:33.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:33.515 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:33.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.515 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:33.515 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 255.84864036499999, 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:33.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:33.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 135.700666625, 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:33.520 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:33.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:33.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:33.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:33.525 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:33.525 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:33.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:33.586 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:33.586+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:33.659 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:33.659 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:33.659 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:33.660 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:33.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:33.686 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:33.687+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:33.790 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:33.790+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:33.804 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:33.805 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:33.805 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:33.805 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:33.805 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:33.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:33.899 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:33.899+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:33.991 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:33.991+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:34.043 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 275.17320534599997, 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 273.99824901400001, 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 273.99728657999998, 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 273.99649657700002, 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 273.96184271599998, 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 273.96115321899998, 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 273.96047986999997, 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 273.888046755, 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 273.13005956299997, 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 272.88140924200002, 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.093 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:34.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:34.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 272.88016385499998, 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:34.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 272.87910434600002, 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:34.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:34.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 272.84625595099999, 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.109 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:34.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.113 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 272.84517082000002, 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:34.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:34.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:34.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 272.84415380199999, 2024-06-03T05:28:34.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:34.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.124 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 272.82437028499999, 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:34.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:34.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 268.699762292, 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.130 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:34.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:34.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 268.698942628, 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:34.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 268.69824167899998, 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 262.29118794700003, 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.150 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:34.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:34.151 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:34.152+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:34.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '18 of 30' 2024-06-03T05:28:34.253 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:34.253+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:34.324 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:34.325 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:34.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:34.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:28:34.378 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:34.378+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:34.454 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:34.454+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:34.467 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:34.467 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:34.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:34.569 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:34.568+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:34.594 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:34.594 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:34.657 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:34.657 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:34.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:34.657 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:34.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 255.78587565300001, 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:34.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 135.667916507, 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:34.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:34.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:34.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:34.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:34.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:34.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:34.666 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:34.667 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:34.687 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:34.687 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:34.688 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:34.687+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:34.757 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:34.757+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:34.824 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:34.824 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:34.833 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:34.833 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:34.834 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:34.878 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:34.876+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:34.941 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:34.942 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:34.944 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:34.959 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:34.959+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:35.011 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:35.011 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:35.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:35.037 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:35.037 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:35.039 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:35.082 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:35.082+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:35.133 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:35.133 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:35.137 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:35.161 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:35.161+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:35.175 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:35.176 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:35.185 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:35.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '19 of 30' 2024-06-03T05:28:35.296 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:35.296+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:35.363 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:35.363+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:35.398 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:35.398 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:35.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:28:35.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:35.463 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:35.463+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:35.582 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:35.583+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:35.598 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:35.598 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:35.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:35.691 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:35.691+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:35.767 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:35.767+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:35.782 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:35.782 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:35.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:35.876 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:35.876+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:35.969 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:35.969+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:35.980 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:35.980 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:35.989 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:36.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:36.102 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:36.102+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:36.170 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:36.170+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:36.170 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:36.170 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:36.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:36.220 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '20 of 30' 2024-06-03T05:28:36.271 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:36.271+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:36.372 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:28:36.373 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:36.372+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:36.407 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:36.407 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:36.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:36.473 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:36.473+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:36.574 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:36.574+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:36.590 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:36.591 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:36.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:36.685 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:36.685+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:36.777 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:36.777+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:36.799 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:36.800 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:36.809 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:36.877 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:36.878+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:36.979 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:36.979+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:36.983 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:36.983 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:36.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:37.079 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:37.080+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:37.127 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:37.127 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:37.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:37.138 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:28:37.182 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:37.182+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:37.269 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:37.269 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:37.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:37.280 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:37.280+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:37.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '21 of 30' 2024-06-03T05:28:37.388 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:37.388+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:37.414 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:28:37.414 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:28:37.414 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:28:37.414 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:28:37.414 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:28:37.414 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:28:37.415 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:28:37.415 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-06-03T05:28:37.415 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-06-03T05:28:37.415 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:28:37.415 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:28:37.415 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:28:37.415 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:37.415 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:37.417 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:28:37.417 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-03T05:28:37.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:37.508 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:37.508+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:37.530 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:37.531 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:37.582 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:37.583+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:37.600 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:37.600 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:37.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:37.624 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:37.624 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:37.684 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:37.684+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:37.725 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:37.725 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:37.783 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:37.783 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:37.785 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:37.785+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:37.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:37.821 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:37.822 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:37.825 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:37.886 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:37.886+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:37.931 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:37.931 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:37.932 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:37.933 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:37.933 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:37.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:37.987 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:37.987+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:38.046 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:38.046 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:38.051 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:38.078 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:38.078 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:38.088 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:38.088+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:38.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:38.189 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:38.189+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:38.281 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:38.281 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:38.290 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:38.290+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:38.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:38.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '22 of 30' 2024-06-03T05:28:38.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:28:38.414 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:38.413+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:38.492 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:38.492+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:38.512 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:38.513 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:38.522 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:38.593 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:38.593+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:38.658 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:38.659 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:38.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:38.694 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:38.694+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:38.795 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:38.795+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:38.847 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:38.847 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:38.848 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:38.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:38.896 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:38.896+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:38.900 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:38.897+0000 5dce500 -1 osd.6 35 log_to_monitors true 2024-06-03T05:28:38.931 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:55.573 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:38.936 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:55.578 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:38.972 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:00:55.613 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:38.997 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:38.997+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:39.023 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:14.845 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:39.028 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:14.851 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:39.066 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:14.889 57344== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:39.098 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:39.098+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 267.27741103400001, 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 267.01857923099999, 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 102.946667276, 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 102.856524024, 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:39.116 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:39.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:39.200 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:39.200+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:39.295 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:39.296 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:39.300 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:39.300+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:39.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:39.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '23 of 30' 2024-06-03T05:28:39.401 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:39.401+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:39.465 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:39.466 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:39.466 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:39.479 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:39.523 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:39.523+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:39.604 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:39.604+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:39.626 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:39.626 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:39.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 103.664674189, 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:39.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:39.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:39.631 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 103.429304079, 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:39.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:39.635 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:39.636 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:39.637 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:39.721 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:39.721+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:39.806 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:39.805+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:39.812 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:39.813 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:39.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:39.909 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:39.908+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:28:40.007 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:40.007+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:40.037 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:40.037 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:40.037 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:40.037 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:40.037 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:40.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:40.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:28:40.108 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:40.108+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:40.210 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:40.210+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:40.273 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 262.60657859700001, 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 142.45860485700001, 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:40.282 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:40.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:40.310 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:40.309+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:40.344 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:40.344 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:40.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '24 of 30' 2024-06-03T05:28:40.411 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:40.411+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:40.416 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:40.417 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:40.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:40.445 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:40.446 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:40.543 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:40.543 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:40.544 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:40.542+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:40.597 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:40.597 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:40.597 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:40.598 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:40.598 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:40.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:40.648 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:40.648 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:40.649 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:40.647+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:40.724 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:40.725+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:40.793 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 281.92129629300001, 2024-06-03T05:28:40.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:40.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:40.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 280.74633996099999, 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:40.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:40.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 280.74537752700002, 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:40.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 280.744587524, 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.812 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:40.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.817 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 280.70993366300002, 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:40.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 280.70924416600002, 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:40.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 280.70857081700001, 2024-06-03T05:28:40.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:40.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.833 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 280.63613770199998, 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:40.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:40.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 279.87815051000001, 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.839 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:40.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:40.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 279.629500189, 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.844 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:40.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:40.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 279.62825480200001, 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:40.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:40.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:40.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 279.627195293, 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:40.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.860 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 279.59434689800003, 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:40.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:40.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 279.593261767, 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.866 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:40.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:40.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:40.871 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 279.59224474899997, 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:40.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.876 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 279.57246123200002, 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 275.44785323899998, 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.882 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:40.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 275.44703357499998, 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:40.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:40.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 275.44633262600001, 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:40.892 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:40.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 269.03927889400001, 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:40.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:40.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:40.902 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:40.902 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:40.903 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:40.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:40.904 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:40.903+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:40.958 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:40.958 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:41.004 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:41.003+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:41.010 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:41.105 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:41.104+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:41.144 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:41.144 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:41.144 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:41.145 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:41.148 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:41.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:41.205 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:41.205+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:41.293 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:41.293 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:41.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:41.347 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:41.347+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:41.407 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:41.406+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:41.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '25 of 30' 2024-06-03T05:28:41.438 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:41.438 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:41.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:41.507 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:41.507+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:41.596 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:41.597 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:41.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:41.609 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:41.608+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:41.709 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:41.709+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:41.789 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:41.789 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:41.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:41.810 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:41.810+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:41.911 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:41.910+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:41.955 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:41.955 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:41.964 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:42.012 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:42.013+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:42.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:28:42.114 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:42.114+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:42.141 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:42.141 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:42.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:42.215 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:42.214+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:42.301 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:42.301 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:42.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:42.316 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:42.316+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:42.416 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:42.417+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:42.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:42.486 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:42.486 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:42.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:42.543 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '26 of 30' 2024-06-03T05:28:42.544 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:42.542+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:42.628 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:42.627+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:42.643 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:42.643 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:42.653 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:42.738 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:42.737+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:42.843 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:42.842+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:42.873 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:42.873 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:42.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:42.920 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:42.920+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:43.022 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:43.022+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:43.059 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:43.059 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:43.068 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:43.123 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:43.122+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:43.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:28:43.224 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:43.223+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:43.226 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:43.226 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:43.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:43.324 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:43.325+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:43.407 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:43.407 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:43.419 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:43.419 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:43.429 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:43.440 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:43.439+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:43.505 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:43.506 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:43.527 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:43.527+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:43.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '27 of 30' 2024-06-03T05:28:43.601 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:43.601 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:43.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:43.627 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:43.627+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:43.630 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:43.630 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:43.641 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:28:43.642 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:28:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:28:43.645 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-03T05:28:43.708 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:28:43.728 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:43.728+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:43.747 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:43.747 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:43.753 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:43.753 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:43.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:43.829 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:43.828+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:43.928 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:43.929 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:43.929 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:43.929+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:43.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:44.030 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:44.029+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:44.069 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:44.070 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:44.070 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:44.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:44.133 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:44.133+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:44.165 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:44.165 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:44.214 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:44.244 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:44.244+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:44.252 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:44.252 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:44.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:44.316 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:44.316 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:44.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:44.321 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:44.365 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:44.364+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:44.437 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:44.436+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:44.438 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:44.438 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:44.438 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:44.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:44.537 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:44.537+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:44.586 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 265.715032888, 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.59707374199999, 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:44.595 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:44.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:44.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '28 of 30' 2024-06-03T05:28:44.677 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:44.677+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:44.736 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:44.736 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:44.736 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:44.736 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:44.739 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:44.738+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:44.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:44.841 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:44.841+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:44.925 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:44.925 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:44.925 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:44.925 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:44.925 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:44.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:44.942 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:44.942+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:45.042 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:45.041+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:45.070 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:45.070 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:45.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:45.070 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:45.070 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:45.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:45.169 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:45.168+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:45.245 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:45.245+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:45.262 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:45.262 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:45.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:45.345 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:45.345+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:45.447 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:45.446+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:45.449 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:45.450 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:45.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:45.548 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:45.547+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:45.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '29 of 30' 2024-06-03T05:28:45.649 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:45.649 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:45.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:45.649 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:45.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.650 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:45.650 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:45.650 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 273.82828501099999, 2024-06-03T05:28:45.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:45.650 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:45.650 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:45.650 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:45.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:45.654 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 273.56945320800003, 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:45.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:45.658 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 109.49754125299999, 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:45.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:45.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:45.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:45.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:45.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:45.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:45.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.663 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 109.407398001, 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:45.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:45.668 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:45.669 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:45.649+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:45.670 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:45.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:28:45.755 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:45.753+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:45.850 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:45.849+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:45.866 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:45.866 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:45.866 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:45.866 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:45.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:45.961 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:45.960+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:46.051 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:46.052+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:46.056 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:46.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:46.056 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:46.056 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:46.056 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:46.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:46.153 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:46.153+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:46.254 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:46.253+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:46.266 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:46.266 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:46.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 110.300819487, 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:46.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:28:46.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 110.06544937699999, 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:28:46.271 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:28:46.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.275 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:46.276 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:46.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:46.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:28:46.362 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:46.360+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:46.455 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:46.454+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:46.475 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:46.475 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:46.475 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:46.476 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:46.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:46.572 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:46.571+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:46.655 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:46.655 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:46.657 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:46.656+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:46.658 INFO:tasks.mon_recovery:adding mon b back in 2024-06-03T05:28:46.658 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-03T05:28:46.658 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-06-03T05:28:46.661 INFO:tasks.ceph.mon.b:Started 2024-06-03T05:28:46.661 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-06-03T05:28:46.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:28:46.664 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:28:46.711 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:46.712 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:46.712 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:46.712 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:46.712 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:46.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:46.762 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:46.762 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:46.763 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:46.762+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:46.859 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:46.859+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:46.862 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:46.920 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 269.25413371000002, 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:46.921 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:46.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:46.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:46.925 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 149.10615996999999, 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:46.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:46.929 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:46.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:46.975 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:46.975+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:46.983 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:46.987 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:47.105 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:47.105 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:47.105 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:47.105+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:47.108 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:47.108 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:47.108 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:47.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:47.161 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:47.161+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:47.250 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:47.250 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:47.254 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:47.267 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:47.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:47.296 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:47.295+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:47.363 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:47.363+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:47.377 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":36,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:28:47.413 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:47.414 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:47.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:47.465 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:47.464+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:47.566 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:47.565+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:47.569 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:47.569 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:47.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:47.667 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:47.666+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:47.731 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:47.731 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:47.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:47.771 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:47.770+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:47.871 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:47.871+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:47.877 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:47.877 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:47.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:47.974 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:47.974+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:48.022 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:48.023 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:48.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:48.075 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:48.075+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:48.177 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:48.175+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:48.205 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:48.206 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:48.215 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:48.277 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:48.277+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:48.378 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:48.377+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:48.388 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:48.389 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:48.399 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:48.484 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:48.483+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:48.574 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:48.574 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:48.580 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:48.579+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:48.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:48.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:48.681 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:48.680+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:48.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:28:48.762 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:48.763 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:48.772 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:48.816 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:48.815+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:48.882 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:48.881+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:48.907 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:48.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:49.002 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:49.001+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:49.083 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:49.083+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:49.099 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:49.099 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:49.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:49.184 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:49.184+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:49.254 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:28:49.284 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:49.285+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:49.291 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:49.291 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:49.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:49.385 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:49.385+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:49.486 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:49.486+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:49.496 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:49.496 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:49.508 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:49.529 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:49.529 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:49.627 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:49.627 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:49.627 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:49.626+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:49.684 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:49.684 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:49.689 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:49.689+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:49.693 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:49.723 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:49.724 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:49.819 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:49.820 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:49.820 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:49.819+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:49.823 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:49.860 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:49.860 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:49.860 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:49.860 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:49.860 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:49.872 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:49.913 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:49.912+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:49.920 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:49.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:49.930 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-03T05:28:49.931 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:49.380+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-06-03T05:28:49.932 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:49.396+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-06-03T05:28:49.932 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:49.469+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-06-03T05:28:49.992 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:49.992+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:50.052 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:50.053 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:50.056 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:50.056 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:50.062 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 272.39619757999998, 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:50.063 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:50.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:50.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:50.067 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 152.24822384000001, 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:50.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:50.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:50.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:50.072 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:50.072 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:50.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:50.114 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:50.113+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:50.155 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:28:50.210 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:50.209+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:50.243 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:50.243 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:50.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:50.294 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:50.293+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:50.296 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:50.074+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-06-03T05:28:50.392 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:28:50.393 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:50.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:50.393 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:50.393 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:50.393 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:50.395 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:50.394+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:50.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:50.476 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:50.297+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-06-03T05:28:50.495 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:50.494+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:50.502 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:50.379+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-06-03T05:28:50.595 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:50.406+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-06-03T05:28:50.595 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:50.457+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-06-03T05:28:50.596 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:50.476+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-06-03T05:28:50.596 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:50.545+0000 5dce500 -1 bdev(0x18c06df0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-06-03T05:28:50.633 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 291.76317079699999, 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 290.58821446500002, 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 290.58725203099999, 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 290.58646202800003, 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 290.55180816699999, 2024-06-03T05:28:50.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 290.55111866999999, 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 290.55044532099998, 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 290.47801220600002, 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.72002501399999, 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.47137469299997, 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.47012930599999, 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.46906979699997, 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.436221402, 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.43513627099998, 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.43411925300001, 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.414335736, 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 285.28972774300001, 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 285.28890807900001, 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 285.28820712999999, 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 278.88115339799998, 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:50.737 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:50.737 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:50.632+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:50.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:28:50.739 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:28:50.740 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:28:50.740 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-03T05:28:50.742 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:50.741+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:50.843 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:50.843+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:50.914 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:50.914 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:50.914 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:50.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:50.944 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:50.943+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:51.045 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:51.044+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:51.072 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":39,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:28:51.107 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:51.107 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:51.107 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:51.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:51.146 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:51.145+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:51.247 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:51.246+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:51.268 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:51.268 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:51.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:51.268 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:51.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.268 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:51.268 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:51.268 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 272.39640500500002, 2024-06-03T05:28:51.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.27844585899999, 2024-06-03T05:28:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:51.277 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:51.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:51.348 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:51.347+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:51.417 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:51.417 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:51.417 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:51.417 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:51.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:51.449 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:51.449+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:51.551 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:51.550+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:51.579 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:51.567+0000 14268640 -1 osd.6 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:28:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:51.614 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:51.652 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:51.652+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:51.752 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:51.752+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:51.789 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:51.789 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:51.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:51.789 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:51.789 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:51.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:51.854 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:51.854+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:51.955 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:51.955+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:51.975 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:51.976 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:51.976 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:51.976 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:51.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:52.055 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:52.055+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:52.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:28:52.119 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:52.120 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:52.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:52.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:28:52.174 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:52.172+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:52.257 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:52.256+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:52.295 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:52.295 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:52.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 280.47087900499997, 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:52.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:52.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 280.21204720200001, 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:52.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:52.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 116.140135247, 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:52.304 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:52.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:52.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:52.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:52.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:52.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:52.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:52.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:52.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:52.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:52.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:52.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 116.049991995, 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:52.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:52.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:52.312 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:52.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:52.336 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:52.337 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:52.357 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:52.357+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:52.452 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:52.452 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:52.458 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:52.457+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:52.519 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:52.520 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:52.520 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:52.520 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:52.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:52.572 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:52.570+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:52.587 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:52.587 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:52.660 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:52.659+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:52.741 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:52.741 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:52.748 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:52.749 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:52.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:52.800 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:52.800 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:52.800+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:52.900 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:52.898+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:52.925 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:52.925 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:52.963 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:52.962+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:52.978 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:52.978 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:52.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:53.072 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:53.071+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:53.162 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:53.163 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:53.164 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:53.163+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:53.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:53.264 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:53.264+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:53.346 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:53.346 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:53.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:53.397 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:53.396+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:53.466 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:53.466+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:53.532 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:53.532 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:53.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:53.587 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:53.587+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:53.669 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:53.668+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:53.681 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:53.681 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:53.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:53.696 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:28:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:28:53.770 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:53.770+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:53.808 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:53.809 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:53.813 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:53.870 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:53.870 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:53.871 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:53.870+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:53.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:53.972 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:53.972+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:54.012 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:54.012 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:54.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:54.074 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:54.073+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:54.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:28:54.102 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:28:54.199 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:54.199+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:54.215 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:54.215 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:54.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:54.276 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:54.275+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:54.376 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:54.376+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:54.422 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:54.422 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:54.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:54.477 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:54.476+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:54.565 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:07.843 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:54.656 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:54.656 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:54.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:54.657 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:54.655+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:54.779 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:54.778+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:54.794 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:54.794 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:54.803 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:54.817 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:08.095 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:54.833 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":43,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:28:54.889 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:54.888+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:54.959 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:54.959+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:54.990 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:54.990 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:55.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:55.060 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:55.060+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:55.150 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:55.151 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:55.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:55.161 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:55.160+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:55.262 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:55.261+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:55.354 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:55.355 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:55.363 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:55.362+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:55.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:55.464 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:55.464+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:55.539 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:55.539 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:55.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:55.592 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:55.591+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:55.666 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:55.666+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:55.684 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:55.685 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:55.694 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:55.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:55.787 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:55.787+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:55.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:28:55.868 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:55.868+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:55.869 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:55.869 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:55.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:55.882 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:55.882 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:55.974 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:55.972+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:28:56.026 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:56.026 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:56.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:28:56.080 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:56.079+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:56.113 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:28:56.113 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:28:56.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:28:56.171 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:56.170+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:56.185 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:56.185 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:56.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:28:56.225 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:28:56.226 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:28:56.272 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:56.272+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:28:56.336 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:28:56.336 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:28:56.373 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:56.372+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:56.385 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:28:56.385 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:28:56.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:28:56.450 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:28:56.450 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:28:56.474 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:56.473+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:56.562 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:28:56.575 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:56.575+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:56.579 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:56.579 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:56.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:28:56.672 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:28:56.672 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:28:56.676 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:56.675+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:56.777 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:56.776+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:56.800 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:56.800 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:56.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:56.800 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:56.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.800 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:56.800 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:56.800 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 279.13339483599998, 2024-06-03T05:28:56.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:56.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:28:56.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:56.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:56.804 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 158.98542109600001, 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:56.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:56.808 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:56.809 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:56.809 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:56.809 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:56.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:28:56.814 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:28:56.815 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:28:56.878 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:56.878+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:56.909 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:28:56.909 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:28:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:28:56.913 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:28:56.951 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:56.951 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:56.951 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:56.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:28:56.978 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:56.978+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:57.080 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:57.079+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:57.096 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:57.096 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:57.096 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:57.096 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:57.096 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:57.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:28:57.180 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:57.180+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:57.281 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:57.281+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:57.315 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:28:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 298.44549951599998, 2024-06-03T05:28:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:28:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:28:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:28:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:28:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:28:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:28:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:28:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:28:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:28:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:28:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:28:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:28:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:28:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:28:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 297.27054318400002, 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:28:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:28:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:28:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:28:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:28:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:28:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:28:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:28:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:28:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:28:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:28:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:28:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:28:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:28:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:28:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 297.26958074999999, 2024-06-03T05:28:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:28:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:28:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:57.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:28:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:28:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:28:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:28:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:28:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:28:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:28:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:28:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:28:57.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:28:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:28:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:28:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:28:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:28:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:28:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:28:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:28:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:28:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:28:57.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 297.26879074700003, 2024-06-03T05:28:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:28:57.340 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:28:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:28:57.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:28:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:28:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:28:57.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:28:57.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:28:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:28:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:28:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:28:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:28:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:57.348 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 297.23413688599999, 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.349 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:57.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:28:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:28:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:28:57.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:28:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:28:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:28:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:28:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:28:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:28:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:28:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:28:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:28:57.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:28:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:28:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:28:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:28:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:28:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.357 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 297.23344738899999, 2024-06-03T05:28:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:28:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:28:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:28:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:28:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:28:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:28:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:28:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:28:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:28:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:28:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:28:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:28:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:28:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:28:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:28:57.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:28:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:28:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:28:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:28:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:28:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:28:57.366 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.367 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 297.23277403999998, 2024-06-03T05:28:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:28:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:28:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:28:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:28:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:28:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:28:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:28:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:28:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:28:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:28:57.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:28:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:28:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:28:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:28:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:28:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:28:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:28:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:28:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:28:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:28:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:28:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.376 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 297.16034092500001, 2024-06-03T05:28:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:28:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:28:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:28:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:28:57.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:28:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:28:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:28:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:28:57.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:57.383 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 296.40235373299998, 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:28:57.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 296.15370341200003, 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:28:57.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:28:57.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 296.15245802499999, 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:28:57.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:28:57.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 296.15139851599997, 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:28:57.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 296.118550121, 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:28:57.404 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:28:57.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 296.11746498999997, 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:28:57.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 296.116447972, 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.415 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:28:57.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:28:57.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 296.096664455, 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 291.97205646200001, 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:28:57.425 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:28:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 291.97123679800001, 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:28:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:28:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 291.97053584899999, 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:28:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:28:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:28:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 285.56348211699998, 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:28:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:28:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.445 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:57.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:28:57.447 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:57.446+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:28:57.548 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:57.548+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:57.624 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:57.625 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:57.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:28:57.680 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:57.680+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:57.749 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:57.748+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:57.771 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:57.771 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:57.771 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:57.772 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:57.772 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:57.776 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:28:57.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:28:57.850 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:57.850+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:57.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:28:57.950 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:57.950+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:57.958 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:57.958 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:57.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:57.958 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:28:57.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 279.08709492000003, 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:57.959 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:28:57.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:28:57.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:28:57.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:28:57.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:28:57.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:28:57.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:28:57.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:28:57.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:28:57.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:28:57.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.96913577399999, 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:28:57.965 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:28:57.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.966 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:28:57.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:28:57.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:57.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:28:57.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:28:57.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:28:57.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:28:57.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:28:57.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:28:57.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:28:57.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:28:57.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:28:57.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:28:57.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:28:57.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.970 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:28:57.970 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:28:57.971 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:57.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:28:58.056 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:58.056+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:58.149 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:58.149 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:58.150 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:28:58.150 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:58.153 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:58.153+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:58.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:28:58.254 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:58.254+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:58.355 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:58.355+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:58.358 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:58.358 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:28:58.358 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:28:58.358 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:28:58.358 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:58.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:28:58.389 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:11.667 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:58.448 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:11.726 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:58.456 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:58.456+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:58.528 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:28:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:28:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:28:58.528 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:28:58.529 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:28:58.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:28:58.565 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":47,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:28:58.565 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:58.566+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:58.658 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:58.658+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:58.702 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:58.702 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:58.702 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:58.702 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:58.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:28:58.715 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:11.993 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:28:58.798 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:58.798+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:58.860 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:58.860+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:58.889 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:58.889 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:58.889 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:58.889 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:58.889 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:58.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:28:58.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:28:58.960 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:58.960+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:59.061 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:28:59.062+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:28:59.073 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:59.074 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:59.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:59.074 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:28:59.074 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:59.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:28:59.163 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:59.163+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:59.257 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:59.257 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:59.257 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:59.257 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:59.264 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:59.264+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:59.266 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:28:59.365 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:59.365+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:28:59.440 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:59.440 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:59.440 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:59.441 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:59.441 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:59.450 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:28:59.494 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:28:59.494+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:28:59.568 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:28:59.568+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 287.77105083499998, 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:28:59.592 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:59.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:59.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:28:59.596 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 287.51221903200002, 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:59.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:28:59.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:28:59.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:59.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 123.440307077, 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:28:59.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 123.350163825, 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:28:59.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:28:59.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:28:59.608 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:59.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:28:59.668 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:28:59.669+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:28:59.769 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:28:59.770+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:28:59.784 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:28:59.784 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:59.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:28:59.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:28:59.871 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:28:59.871+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:28:59.970 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:28:59.970 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:28:59.970 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:28:59.971 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:28:59.971 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:28:59.972 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:28:59.972+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:28:59.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:00.072 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:00.073+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:00.160 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:00.160 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 124.199015895, 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:29:00.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:00.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 123.963645785, 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:00.165 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:00.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:29:00.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:29:00.169 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.169 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:00.169 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.169 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.169 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:00.169 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:00.170 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:00.214 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:00.212+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:00.273 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:00.274+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:00.348 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:00.348 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:00.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:00.401 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:00.401+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:00.476 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:00.477+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:00.491 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:00.491 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:00.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:00.586 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:00.586+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:00.677 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:00.677+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:00.681 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 283.01479667299998, 2024-06-03T05:29:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:29:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 162.86682293300001, 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:29:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:29:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:29:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:00.690 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:00.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:00.777 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:00.778+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:00.838 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:00.838 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:00.838 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:00.838 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:00.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:00.878 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:00.879+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:00.891 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 7, 6] 2024-06-03T05:29:00.892 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:29:00.979 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:00.979+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:01.022 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:01.022 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:01.022 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:01.022 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:01.022 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:01.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:01.081 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:01.081+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:01.182 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:01.182+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:01.215 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 302.34633100100001, 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:29:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:29:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 301.17137466899999, 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:29:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 301.17041223500001, 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:29:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 301.16962223199999, 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:29:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 301.13496837100001, 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:29:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:29:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 301.13427887400002, 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 301.13360552500001, 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:29:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 301.06117240999998, 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:29:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:29:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:29:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:29:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:29:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:29:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:29:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 300.30318521800001, 2024-06-03T05:29:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:29:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:29:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 300.054534897, 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 300.05328951000001, 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:29:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:29:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:29:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 300.052230001, 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 300.01938160600002, 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:29:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 300.018296475, 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:29:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 300.01727945699997, 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:29:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:29:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 299.99749594000002, 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:29:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:29:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 295.87288794699998, 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:29:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 295.87206828299998, 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:29:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:29:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:29:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 295.87136733400001, 2024-06-03T05:29:01.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:01.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:29:01.318 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 289.464313602, 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:29:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:29:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.322 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.322 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:01.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:01.324 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:01.324+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:01.425 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:01.425+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:01.457 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:01.457 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:01.457 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:01.457 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:01.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:01.526 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:01.526+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:01.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:29:01.599 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:01.599 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:01.599 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:01.599 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:01.599 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:01.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:01.652 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:01.652+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:01.728 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:01.728+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:01.745 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:01.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 282.87408392399999, 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:29:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.75612477799999, 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:29:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:29:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:01.754 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:01.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:01.829 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:01.829+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:01.928 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:01.928 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:01.930 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:01.930+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:01.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:02.031 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:02.031+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:02.113 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:02.113 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:02.113 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:02.113 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:02.113 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:02.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:02.165 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:02.164+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:02.232 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:02.232+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:02.298 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:02.298 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:02.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:02.298 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:29:02.298 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:02.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:02.333 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:02.333+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:02.434 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:02.434+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:02.451 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:02.451 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:02.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:02.535 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:02.535+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:02.598 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:02.598 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:02.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:02.653 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:02.653+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:02.737 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:02.737+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:02.741 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:02.741 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:02.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:02.742 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:29:02.742 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:02.750 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:02.838 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:02.838+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:02.925 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:02.925 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:02.925 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:02.925 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:02.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:02.976 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:02.976+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:03.039 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:03.040+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:03.113 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:03.114 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:03.114 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:03.114 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:03.114 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:03.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:03.140 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:03.141+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:03.242 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:03.241+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:03.306 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:03.306 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 291.48530861299997, 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:29:03.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:03.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 291.22647681000001, 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:03.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:29:03.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.316 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:29:03.316 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:03.316 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 127.154564855, 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:03.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:03.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:29:03.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:29:03.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:03.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:29:03.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:29:03.320 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.320 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 127.064421603, 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:29:03.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.325 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:03.325 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:03.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:03.371 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:03.371+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:03.444 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:03.443+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:03.460 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:03.460 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:03.460 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:03.460 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:03.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:03.545 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:03.544+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:03.646 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:03.646+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:03.646 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:03.646 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:03.646 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:03.646 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:03.646 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:03.655 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:03.747 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:03.747+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:03.834 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:03.834 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:03.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:03.834 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:03.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.834 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 127.872784852, 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:29:03.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:29:03.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:03.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.841 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:03.841 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:03.841 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 127.637414742, 2024-06-03T05:29:03.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:29:03.841 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:03.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:03.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:03.846 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:03.846 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:03.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:03.848 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:03.847+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:03.949 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:03.949+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:04.043 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:04.043 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:04.043 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:04.043 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:04.050 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:04.049+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:04.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:04.151 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:04.150+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:04.238 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:04.238 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:04.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:04.292 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:04.292+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:04.352 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:04.352+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:04.389 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:04.389 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 286.72296364499999, 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:04.390 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:04.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:04.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:29:04.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 166.574989905, 2024-06-03T05:29:04.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:04.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:29:04.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:04.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:04.398 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:04.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:04.453 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:04.452+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:04.546 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:04.546 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:04.546 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:04.554 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:04.553+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:04.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:04.655 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:04.654+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:04.729 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:04.729 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:04.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:04.756 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:04.756+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:04.857 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:04.856+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:04.962 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:04.962 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:04.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:04.963 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:04.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.963 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:29:04.963 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:04.963 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 306.09314707599998, 2024-06-03T05:29:04.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:29:04.963 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:04.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:04.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:04.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:04.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 304.91819074400001, 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:04.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:29:04.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:04.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:04.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 304.91722830999998, 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:04.975 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:29:04.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:29:04.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 304.91643830700002, 2024-06-03T05:29:04.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:04.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:29:04.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:04.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:29:04.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 304.88178444599998, 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:04.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:04.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:29:04.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:04.991 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 304.88109494899999, 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:29:04.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:29:04.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 304.88042159999998, 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:04.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:04.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:04.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:29:05.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:29:05.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 304.80798848500001, 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:29:05.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:29:05.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:05.008 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 304.05000129299998, 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:29:05.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:05.014 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 303.80135097200002, 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:29:05.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:29:05.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 303.80010558499998, 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:29:05.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.025 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 303.79904607600002, 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.026 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 303.76619768099999, 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:29:05.031 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:29:05.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:29:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:29:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:29:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:29:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:29:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:29:05.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 303.76511255000003, 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:29:05.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:29:05.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:29:05.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:29:05.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 303.764095532, 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:29:05.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:29:05.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 303.74431201499999, 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:29:05.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:29:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:29:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:29:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:29:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 299.61970402200001, 2024-06-03T05:29:05.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:29:05.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:29:05.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 299.618884358, 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:29:05.063 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:29:05.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:29:05.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:29:05.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 299.61818340899998, 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:05.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:29:05.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:29:05.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:29:05.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 293.21112967699997, 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.075 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:29:05.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:29:05.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:29:05.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.079 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:05.080 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:04.960+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:05.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:05.122 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:05.121+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:05.184 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:05.183+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:05.189 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:05.190 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:05.254 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:05.254 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:05.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:05.283 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":56,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:29:05.284 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:05.284 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:05.285 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:05.285+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:05.379 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:05.379 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:05.385 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:05.384+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:05.437 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:05.437 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:05.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:05.476 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:05.476 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:05.479 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:05.508 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:05.507+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:05.590 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:05.589+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:05.590 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:05.640 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:05.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 286.76872983700002, 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:29:05.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 166.65077069099999, 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:29:05.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:05.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.648 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:05.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:05.649 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:05.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:05.685 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:05.685 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:05.693 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:05.692+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:05.778 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:05.779 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:05.783 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:05.811 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:05.810+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:05.826 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:05.827 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:05.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:05.893 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 7, 6] 2024-06-03T05:29:05.893 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-06-03T05:29:05.893 INFO:tasks.ceph.osd.4:Restarting daemon 2024-06-03T05:29:05.893 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-06-03T05:29:05.920 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:05.919+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:05.921 INFO:tasks.ceph.osd.4:Started 2024-06-03T05:29:05.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:05.991 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:05.991+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:06.011 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:06.011 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:06.016 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-06-03T05:29:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:29:06.019 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-06-03T05:29:06.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:06.091 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:06.092+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:06.192 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:06.193+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:06.198 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:06.198 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:06.209 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:06.294 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:06.293+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:06.343 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:06.343 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:06.353 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:06.397 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:06.397+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:06.488 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:06.488 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:06.496 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:06.495+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:06.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:06.597 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:06.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:06.670 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:06.671 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:06.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:06.725 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:06.725+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:06.798 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:06.798+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:06.813 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:06.813 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:06.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:06.899 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:06.898+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:07.000 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:07.001+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:07.004 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:07.004 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:07.014 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:07.102 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:07.102+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:07.200 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:07.200 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:07.202 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:07.201+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:07.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:07.303 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:07.302+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:07.386 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:07.386 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:07.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:07.443 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:07.443+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:07.504 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:07.503+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:07.534 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:07.534 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:07.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:07.628 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:07.628+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:07.706 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:07.705+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:07.716 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:07.717 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:07.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:07.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:29:07.810 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:07.810+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:07.907 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:07.906+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:07.912 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:07.912 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:07.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:08.011 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:08.011+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:08.060 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:08.060 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:08.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:08.112 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:08.112+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:08.215 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:08.214+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:08.256 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:08.257 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:08.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:08.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:29:08.343 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:08.343+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:08.382 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:29:08.404 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:08.405 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:08.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:08.419 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:08.418+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:08.484 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:08.484 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:08.521 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:08.521+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:08.572 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:08.572 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:08.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:08.594 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:08.595 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:08.628 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:08.627+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:08.721 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:08.721 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:08.724 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:08.724+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:08.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:08.736 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:08.736 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:08.827 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:08.827+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:08.832 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:08.833 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:08.904 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:08.904 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:08.912 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:08.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:09.027 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:09.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:09.028 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:09.027+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:09.031 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:09.031+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:09.081 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:09.082 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:09.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:09.135 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:09.134+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:09.136 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:09.136 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:09.171 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:29:09.188 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-06-03T05:29:09.188 INFO:tasks.mon_recovery:removing mon c 2024-06-03T05:29:09.189 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-03T05:29:09.189 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:29:09.234 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:09.231+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:09.239 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:09.239 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:09.243 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:09.287 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 290.416190013, 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:09.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:29:09.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 170.298230867, 2024-06-03T05:29:09.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:09.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:09.296 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:09.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:09.299 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:29:09.288+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 57339) UID: 0 2024-06-03T05:29:09.302 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:29:09.289+0000 9f9e640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-06-03T05:29:09.331 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:09.331+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:09.432 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:09.432+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:09.491 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:09.491 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:09.491 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:09.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:09.533 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:09.534+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:09.651 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:09.650+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:09.668 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:09.669 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:09.669 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:09.669 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:09.669 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:09.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:09.736 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:09.736+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:09.838 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:09.838+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:09.865 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:09.865 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:29:09.865 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:09.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:09.912 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:45.734 57344== 2024-06-03T05:29:09.912 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:45.734 57344== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:29:09.924 INFO:tasks.ceph.mon.c.smithi038.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:29:09.950 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:09.943+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:10.039 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:10.039 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:10.039 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:10.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:10.052 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:10.050+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:10.140 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:10.140+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:10.225 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:10.225 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:10.225 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:10.225 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:10.225 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:10.234 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:10.241 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:10.240+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:10.342 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:10.342+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:10.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:29:10.409 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:10.409 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:10.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:10.409 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:29:10.409 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:10.419 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:10.462 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:10.462+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:10.544 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:10.543+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:10.604 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:10.604 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:10.604 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:10.604 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:10.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:10.656 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:10.656+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:10.745 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:10.745 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:10.746 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:10.745+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:10.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:10.847 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:10.847+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:10.938 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 299.11788747200001, 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:10.939 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:10.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:29:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:10.943 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 298.85905566899999, 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:10.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:10.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:10.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:10.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:29:10.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:29:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:29:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:29:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:29:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:29:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:29:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:29:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 134.787143714, 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:10.952 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 134.69700046200001, 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:10.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:10.956 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:10.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:10.959 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:10.957+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:11.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:11.065 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:11.064+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:11.117 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-06-03T05:29:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:29:11.119 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-06-03T05:29:11.135 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:11.135 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:11.135 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:11.135 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:11.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:11.189 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:11.189+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:11.245 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:29:11.279 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:11.279 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:11.279 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:11.279 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:11.280 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:11.280 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:11.280+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:11.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:11.374 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:11.374+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:11.461 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:11.461+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:11.478 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:11.478 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:11.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 135.51704582400001, 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:29:11.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:11.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:29:11.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:29:11.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:29:11.483 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 135.28167571399999, 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:11.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:29:11.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:11.488 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:11.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:11.501 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:11.502 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:11.561 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:11.560+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:11.597 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:11.597 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:11.639 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:11.639 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:11.639 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:11.639 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:11.649 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:11.662 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:11.662+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:11.701 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:11.702 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:11.763 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:11.763+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:11.830 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:11.831 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:11.831 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:11.832 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:11.834 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:11.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:11.883 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:11.883+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:11.931 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:11.932 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:11.966 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:11.966+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:12.032 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:12.032 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:12.041 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:12.044 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:12.044 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:12.066 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:12.066+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:12.139 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:12.139 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:12.143 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:12.167 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:12.166+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:12.179 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:12.179 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:12.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:12.273 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:12.272+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:12.361 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:12.361 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:12.369 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:12.369+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:12.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:12.470 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:12.470+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:12.546 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:12.546 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:12.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:12.599 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:12.599+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:12.672 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:12.672+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:12.688 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:12.688 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:12.697 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:12.782 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:12.782+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:12.873 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:12.873 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:12.874 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:12.874+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:12.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:12.975 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:12.975+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:13.059 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:13.059 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:13.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:13.111 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:13.110+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:13.177 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:13.177+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:13.243 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:13.244 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:13.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:13.296 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:13.296+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:13.338 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:07.355 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:13.379 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:13.379+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:13.388 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:13.388 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:13.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:13.403 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:07.420 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:13.480 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:13.480+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:13.536 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:13.536 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:13.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:13.591 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:13.591+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:13.682 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:13.682+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:13.692 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:13.692 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:13.702 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:13.783 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:13.783+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:13.884 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:13.884+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:13.886 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:13.886 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:13.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:13.976 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:13.971+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:29:13.985 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:13.985+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:14.030 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:14.030 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:14.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:14.124 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:14.124+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:14.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:29:14.188 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:14.188+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:14.230 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:14.231 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:14.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:14.326 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:14.326+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:14.387 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:14.387+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:14.411 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:08.428 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:14.414 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:14.414 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:14.424 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:14.488 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:14.488+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:14.589 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:14.589+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:14.599 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:14.599 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:14.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:14.690 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:14.690+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:14.742 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:14.743 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:14.753 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:14.782 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:08.799 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:14.791 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:14.791+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:14.887 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:14.887 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:14.892 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:14.892+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:14.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:14.994 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:14.994+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:15.075 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:15.075 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:15.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:15.128 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:15.127+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:15.140 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:09.156 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:15.198 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:15.198+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:15.263 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:15.263 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:15.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:29:15.298 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-06-03T05:29:15.298 INFO:tasks.ceph.mon.c:Stopped 2024-06-03T05:29:15.299 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 2 2024-06-03T05:29:15.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:29:15.326 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:15.327+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:15.399 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:15.399+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:15.416 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:15.416 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:15.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:15.432 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:09.449 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:15.499 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:15.499+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:15.600 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:15.600+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:15.607 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:15.608 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:15.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:15.702 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:15.702+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:15.732 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:09.749 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:15.759 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:15.759 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:15.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:15.802 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:15.802+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:15.903 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:15.903+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:15.946 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:15.946 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:15.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:16.004 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:16.005+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:16.005 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:29:16.037 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:10.054 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:16.118 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:16.118+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:16.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:16.133 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:16.133 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:16.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:16.214 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:16.212+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:16.307 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:16.307+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:16.313 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:29:16.313 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:29:16.313 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:29:16.313 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:29:16.314 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:29:16.314 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:29:16.314 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-06-03T05:29:16.314 INFO:teuthology.orchestra.run.smithi064.stderr:config diff get 2024-06-03T05:29:16.314 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-06-03T05:29:16.314 INFO:teuthology.orchestra.run.smithi064.stderr:config help [] 2024-06-03T05:29:16.314 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-06-03T05:29:16.314 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:29:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:29:16.316 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-06-03T05:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:16.328 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:16.346 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:10.363 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:16.408 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:16.408+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:16.478 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:16.478 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:16.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:16.509 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:16.509+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:16.610 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:16.610+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:16.638 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:10.655 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:16.660 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:16.660 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:16.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:16.714 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:16.713+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:16.812 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:16.812+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:16.844 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:16.844 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:16.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:16.913 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:16.914+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:16.933 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:10.950 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:16.989 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:16.989 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:16.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:17.014 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:17.015+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:17.116 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:17.116+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:17.135 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:17.135 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:17.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:17.216 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:17.216+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:17.288 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:17.288 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:17.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:17.317 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:17.317+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:17.418 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:17.418+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:17.457 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:11.474 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:17.471 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:17.471 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:17.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:17.513 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:11.530 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:17.518 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:17.518+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:17.616 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:17.616 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:17.619 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:17.619+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:17.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:17.721 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:17.721+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:17.759 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:17.759 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:17.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:17.853 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:17.853+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:17.922 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:17.922+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:17.961 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:17.961 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:17.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:18.055 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:18.055+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:18.124 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:18.124+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:18.148 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:18.148 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:18.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:18.224 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:18.224+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:18.325 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:18.326+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:18.331 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:18.331 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:18.341 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:18.427 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:18.427+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:18.514 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:18.514 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:18.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:18.527 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:18.527+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:18.628 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:18.628+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:18.700 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:18.701 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:18.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:18.729 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:18.729+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:18.830 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:18.830+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:18.844 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:18.844 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:18.854 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:18.939 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:18.939+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:19.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:29:19.028 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:19.028 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:19.032 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:19.032+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:19.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:19.133 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:19.133+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:19.214 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:19.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:19.235 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:19.235+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:19.336 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:19.336+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:19.359 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:19.359 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:19.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:19.436 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:19.436+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:19.522 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:19.523 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:19.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:19.537 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:19.537+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:19.638 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:19.638+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:19.670 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:19.670 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:19.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:19.739 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:19.739+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:19.840 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:19.840+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:19.855 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:19.856 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:19.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:19.941 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:19.941+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:19.998 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:19.998 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:20.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:20.042 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:20.043+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:20.144 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:20.144+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:20.181 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:20.181 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:20.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:20.244 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:20.245+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:20.326 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:20.326 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:20.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:20.379 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:20.379+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:20.446 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:20.446+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:20.469 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:20.469 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:20.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:20.547 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:20.547+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:20.648 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:20.648+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:20.654 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:20.654 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:20.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:20.749 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:20.749+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:20.806 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:20.806 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:20.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:20.850 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:20.850+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:20.951 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:20.951+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:20.989 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:20.989 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:20.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:21.052 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:21.052+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:21.153 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:21.153+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:21.173 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:21.173 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:21.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:21.254 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:21.254+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:21.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:21.355 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:21.355+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:21.357 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:21.357 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:21.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:21.456 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:21.456+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:21.541 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:21.541 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:21.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:21.556 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:21.556+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:21.657 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:21.657+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:21.722 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:21.723 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:21.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:21.758 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:21.758+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:21.859 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:21.859+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:21.866 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:21.867 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:21.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:21.960 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:21.960+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:22.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:22.062 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:22.061+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:22.162 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:22.163+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:22.194 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:22.194 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:22.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:22.263 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:22.264+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:22.364 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:22.364+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:22.378 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:22.379 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:22.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:22.465 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:22.465+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:22.521 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:22.521 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:22.530 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:22.574 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:22.574+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:22.664 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:22.665 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:22.666 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:22.667+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:22.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:22.703 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:29:22.703 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:29:22.703 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:29:22.703 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:29:22.704 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:29:22.704 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:29:22.704 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:29:22.704 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:29:22.704 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation histogram block [] [] 2024-06-03T05:29:22.705 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator score block 2024-06-03T05:29:22.705 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-06-03T05:29:22.705 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:29:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:29:22.707 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-06-03T05:29:22.768 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:22.768+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:22.789 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:16.806 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:22.809 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:22.810 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:22.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:22.904 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:22.904+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:22.969 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:22.969+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:22.994 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:22.995 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:23.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:23.070 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:23.070+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:23.170 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:23.171+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:23.180 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:23.181 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:23.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:23.271 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:23.272+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:23.324 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:23.324 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:23.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:23.373 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:23.372+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:23.468 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:23.468 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:23.473 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:23.473+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:23.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:23.574 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:23.574+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:23.651 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:23.651 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:23.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:23.674 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:23.674+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:23.775 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:23.776+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:23.795 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:23.795 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:23.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:23.876 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:23.876+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:23.941 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:23.942 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:23.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:23.994 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:23.993+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:24.078 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:24.078+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:24.127 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:24.127 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:24.136 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:24.180 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:24.180+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:24.280 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:24.280+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:24.281 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:24.281 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:24.291 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:24.381 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:24.381+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:24.466 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:24.466 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:24.475 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:24.482 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:24.482+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:24.546 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:29:24.561 INFO:tasks.mon_recovery.ceph_manager:quorum is size 2 2024-06-03T05:29:24.561 INFO:tasks.mon_recovery:causing some monitor log activity 2024-06-03T05:29:24.561 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '1 of 30' 2024-06-03T05:29:24.611 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:24.611+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:24.626 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:24.626 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:24.631 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:41.273 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:24.636 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:24.639 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:41.281 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:24.684 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:24.684+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:24.690 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:41.333 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:24.696 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:37.974 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:24.698 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:37.976 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:24.762 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:38.040 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:24.781 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:24.781 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:24.785 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:24.785+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:24.791 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:24.886 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:24.887+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:24.969 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:24.970 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:24.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:24.987 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:24.987+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:25.056 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:25.056 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:25.088 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:25.088+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:25.154 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:25.154 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:25.189 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:25.189+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:25.205 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:25.205 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:25.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:25.205 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:25.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.205 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:25.205 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 313.38382509000002, 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:29:25.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:25.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:25.209 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 313.124993287, 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:25.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:29:25.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:29:25.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:25.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:29:25.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:29:25.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:25.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:29:25.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:29:25.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 149.053081332, 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:29:25.215 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:25.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:25.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 148.96293807999999, 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:25.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.222 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:25.223 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.223 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.223 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:25.223 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:25.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:25.272 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:25.272 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:25.290 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:25.290+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:25.370 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:25.370 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:25.374 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:25.391 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:25.391+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:25.402 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:25.402 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:25.402 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:25.402 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:25.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:25.466 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:25.467 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:25.492 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:25.492+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:25.561 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:25.626 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:25.627 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:25.627 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:25.627 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:25.627 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:25.627 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:25.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:25.629 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:25.629+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:25.665 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:25.665 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:25.669 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:25.669 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:29:25.730 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:25.730+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:25.808 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:25.808 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:25.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:25.808 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:25.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.808 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:25.808 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:25.809 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 149.84634085799999, 2024-06-03T05:29:25.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:29:25.809 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:25.809 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:25.809 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:25.809 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:25.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:29:25.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:25.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:29:25.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:29:25.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:25.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:29:25.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:29:25.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:25.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:29:25.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:29:25.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:25.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:29:25.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:29:25.815 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.815 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:25.815 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 149.610970748, 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:25.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:29:25.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:29:25.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:25.820 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:25.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '2 of 30' 2024-06-03T05:29:25.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:25.830 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:25.831+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:25.933 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:25.933+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:26.010 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:26.010 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:26.010 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:26.011 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:26.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:26.034 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:26.033+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:26.135 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:26.135+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:26.193 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:26.194 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:26.194 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:26.194 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:26.194 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:26.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:26.243 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:26.244+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:26.336 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:26.336+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:26.404 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 308.73463820900002, 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:29:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:29:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 188.586664469, 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:29:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:29:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:29:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:26.413 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:26.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:26.437 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:26.437+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:26.538 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:26.538+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:26.550 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:26.550 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:26.550 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:26.550 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:26.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:26.639 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:26.639+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:26.695 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:26.695 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:26.695 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:26.695 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:26.695 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:26.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:26.740 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:26.740+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:26.841 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:26.841+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:26.891 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:26.891 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:26.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:26.891 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 328.0215819, 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:29:26.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:29:26.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 326.84662556799998, 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.897 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:29:26.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 326.84566313400001, 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:29:26.902 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:29:26.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:26.907 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 326.84487313099999, 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:26.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:29:26.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.912 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 326.81021927, 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:29:26.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:29:26.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 326.80952977300001, 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.918 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:29:26.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:29:26.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 326.808856424, 2024-06-03T05:29:26.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:26.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 326.73642330899997, 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:29:26.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:29:26.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 325.978436117, 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:29:26.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:29:26.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 325.72978579599999, 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.939 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:29:26.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 325.728540409, 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:29:26.944 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:29:26.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.949 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 325.72748089999999, 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.950 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:29:26.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:29:26.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 325.69463250500002, 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:29:26.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:29:26.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 325.69354737399999, 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:29:26.960 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:29:26.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:29:26.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.965 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 325.69253035600002, 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:29:26.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:29:26.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:29:26.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 325.67274683900001, 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 321.54813884599997, 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:29:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.982 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 321.54731918200002, 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:29:26.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:29:26.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 321.546618233, 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:26.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:29:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:26.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 315.139564501, 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:26.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:29:26.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:26.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:29:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:29:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:26.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:26.997 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:26.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '3 of 30' 2024-06-03T05:29:26.999 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:26.999+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:27.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:27.099 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:27.100+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:27.135 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:27.135 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:27.136 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:27.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:27.201 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:27.201+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:27.283 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:27.283 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:27.284 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:27.284 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:27.284 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:27.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:27.302 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:27.301+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:27.402 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:27.402+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:27.431 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:27.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 308.55970189800001, 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:27.432 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:27.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:27.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:29:27.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 188.44174275200001, 2024-06-03T05:29:27.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:27.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:27.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:29:27.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:27.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:27.441 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:27.441 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:27.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:27.503 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:27.503+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:27.581 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:27.581 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:27.581 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:27.581 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:27.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:27.632 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:27.632+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:27.670 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:29:27.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:29:27.703 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:27.703+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:27.708 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:27.765 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:27.765 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:27.766 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:27.766 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:27.775 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:27.804 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:27.804+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:27.824 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:29:27.824 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation histogram block [] [] 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator score block 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-06-03T05:29:27.825 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:29:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:29:27.827 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-06-03T05:29:27.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '4 of 30' 2024-06-03T05:29:27.905 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:27.905+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:27.932 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:27.932 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:29:27.932 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:27.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:27.946 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:27.946 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:28.007 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:28.007+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:28.044 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:28.044 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:28.103 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:28.103 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:28.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:28.141 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:28.141 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:28.142 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:28.141+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:28.161 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:22.178 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:28.165 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:22.182 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:28.208 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:28.208+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:28.240 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:28.241 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:28.280 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:28.280 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:28.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:28.334 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:28.334+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:28.387 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:28.410 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:28.410+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:28.430 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:28.431 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:28.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:28.481 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:28.481 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:28.525 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:28.525+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:28.595 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:28.595 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:28.613 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:28.613+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:28.628 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:28.628 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:28.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:28.691 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:28.692 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:28.696 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:28.713 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:28.714+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:28.814 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:28.814 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:28.814 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:28.814+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:28.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:28.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '5 of 30' 2024-06-03T05:29:28.940 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:28.940+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:28.987 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:28.987 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:28.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:29.011 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:29:29.029 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:29.029+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:29.116 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:29.117+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:29.145 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:29.145 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:29.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:29.217 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:29.218+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:29.318 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:29.319+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:29.330 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:29.330 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:29.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:29.403 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:46.045 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:29.405 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:46.047 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:29.419 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:29.419+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:29.430 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:01:46.072 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:29.459 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:42.737 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:29.461 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:42.740 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:29.477 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:42.755 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:29.520 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:29.520+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:29.529 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:29.529 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:29.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:29.624 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:29.624+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:29.717 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:29.717 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:29.722 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:29.723+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:29.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:29.823 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:29.824+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:29.905 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:29.906 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:29.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:29.924 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:29.924+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:29.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '6 of 30' 2024-06-03T05:29:30.065 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:30.065+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:30.110 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:30.111 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:30.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:30.162 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:30.162+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:30.226 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:30.226+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:30.255 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:30.255 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:30.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:30.327 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:30.327+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:30.397 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:30.398 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:30.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:30.428 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:30.429+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:30.530 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:30.530+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:30.545 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:30.546 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:30.558 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:30.630 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:30.631+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:30.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:29:30.731 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:30.732 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:30.732 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:30.732+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:30.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:30.833 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:30.833+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:30.915 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:30.915 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:30.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:30.934 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:30.934+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:30.943 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:30.943 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:31.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:29:31.034 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:31.034+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:31.038 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:31.038 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:31.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '7 of 30' 2024-06-03T05:29:31.114 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:31.114 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:31.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:31.152 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:31.152 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:31.166 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:31.166+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:31.253 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:31.253 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:31.254 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:31.253+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:31.302 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:31.302 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:31.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:31.336 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:31.336+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:31.437 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:31.437+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:31.449 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:31.449 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:31.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:31.513 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:31.537 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:31.538+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:31.627 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:31.627 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:31.638 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:31.638+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:31.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:31.651 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:31.651 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:31.739 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:31.739+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:31.769 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:31.770 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:31.788 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:31.788 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:31.788 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:31.788 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:31.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:31.863 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:31.863 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:31.864 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:31.863+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:31.867 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:31.941 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:31.941+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:31.974 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:31.974 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:31.974 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:31.974 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:31.974 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:31.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:32.042 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:32.043+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:32.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '8 of 30' 2024-06-03T05:29:32.123 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:29:32.123 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:32.133 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:32.163 INFO:tasks.ceph.osd.4.smithi064.stderr:==00:00:00:26.180 58893== Warning: unimplemented fcntl command: 1038 2024-06-03T05:29:32.176 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:32.175+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:32.192 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:29:32.249 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:32.249+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:32.337 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:32.337 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:32.337 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:32.338 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:32.344 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:32.344+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:32.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:32.445 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:32.446+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:32.519 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:32.519 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:32.519 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:32.519 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:32.520 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:32.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:32.546 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:32.546+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:32.647 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:32.647+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 320.89090518, 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:29:32.712 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:32.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:29:32.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:32.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:32.716 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 320.63207337699998, 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:32.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:29:32.720 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 156.56016142199999, 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:32.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 156.47001817, 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:29:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:32.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:32.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:32.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:29:32.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:29:32.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:32.728 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:32.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:32.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:32.728 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:32.728 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:32.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:32.771 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:32.771+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:32.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:32.849 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:32.849+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:32.904 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:32.904 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:32.904 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:32.905 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:32.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:32.956 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:32.955+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:33.050 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:33.050+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:33.080 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '9 of 30' 2024-06-03T05:29:33.088 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:33.088 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:33.089 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:33.089 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:33.089 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:33.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:33.150 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:33.151+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:33.251 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:33.252+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:33.288 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:33.288 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:33.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:33.288 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 157.32684010899999, 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:33.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:29:33.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 157.091469999, 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:29:33.293 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:33.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:33.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:29:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:29:33.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.298 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:33.298 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.298 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.298 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:33.298 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:33.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:33.352 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:33.352+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:33.458 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:33.458+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:33.472 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:33.473 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:33.473 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:33.473 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:33.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:33.555 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:33.555+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:33.656 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:33.656+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:33.657 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:33.657 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:33.657 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:33.658 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:33.658 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:33.667 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:33.757 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:33.757+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:33.807 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 316.14108685899998, 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:33.808 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:33.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:29:33.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 195.99311311899999, 2024-06-03T05:29:33.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:33.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:33.816 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:33.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:33.863 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:33.863+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:33.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:29:33.954 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:33.954 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:33.954 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:33.959 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:33.959+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:33.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:34.060 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:34.060+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:34.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '10 of 30' 2024-06-03T05:29:34.140 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:34.140 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:34.141 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:34.141 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:34.148 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:34.148 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:34.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:34.192 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:34.191+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:34.209 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:29:34.250 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:34.250 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:34.261 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:34.261+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:34.346 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:34.347 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 335.50657177300002, 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:29:34.376 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:29:34.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:29:34.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:34.381 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 334.331615441, 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:29:34.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:29:34.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 334.33065300700002, 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:29:34.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.391 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 334.329863004, 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:29:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:29:34.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 334.29520914300002, 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.397 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.398 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:29:34.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.399 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:34.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:29:34.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.403 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 334.29451964600003, 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:29:34.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:29:34.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 334.29384629700002, 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.409 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:29:34.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 334.22141318199999, 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:29:34.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:29:34.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 333.46342599000002, 2024-06-03T05:29:34.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:34.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:29:34.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.424 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 333.21477566900001, 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:29:34.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:29:34.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 333.21353028200002, 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.430 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:29:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:29:34.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 333.21247077300001, 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:29:34.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 333.17962237799998, 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:29:34.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:29:34.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 333.17853724700001, 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:34.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:29:34.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 333.17752022899998, 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 333.15773671199997, 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:29:34.455 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:29:34.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:29:34.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 329.03312871899999, 2024-06-03T05:29:34.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:29:34.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:34.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 329.03230905499998, 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 329.03160810600002, 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:29:34.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:29:34.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 322.62455437400001, 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:29:34.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:34.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:34.479 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:34.480 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:34.375+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:34.481 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:34.481 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:34.481 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:34.482 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:34.480+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:34.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:34.543 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:34.544 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:34.581 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:34.581+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:34.642 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:34.642 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:34.675 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:34.675 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:34.676 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:34.676 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:34.682 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:34.681+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:34.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:34.759 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:34.760 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:34.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:34.807 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:34.806+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:34.861 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:34.861 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:34.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:34.914 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:34.914+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:34.985 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:34.985+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:35.020 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:35.020 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:35.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:35.051 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:29:35.051 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:29:35.051 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:29:35.051 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:29:35.051 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:29:35.052 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:29:35.052 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:29:35.052 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-06-03T05:29:35.052 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-06-03T05:29:35.052 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:29:35.052 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:29:35.052 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:29:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:29:35.055 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-06-03T05:29:35.096 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:35.096+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:35.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '11 of 30' 2024-06-03T05:29:35.186 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:35.187 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:35.187 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:35.187+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:35.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:35.288 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:35.288+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:35.334 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:29:35.390 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:35.390 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:35.390 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:35.390+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:35.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:35.491 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:35.491+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:35.592 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:35.592 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:35.592+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:35.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:35.693 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:35.693+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:35.743 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:35.743 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:35.753 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:35.797 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:35.796+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:35.886 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:35.886 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:35.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:35.896 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:35.895+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:35.999 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:35.998+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:36.037 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:36.037 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:36.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:36.154 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:36.153+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:36.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '12 of 30' 2024-06-03T05:29:36.196 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:36.197+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:36.233 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:36.234 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:36.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:36.331 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:36.330+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:36.398 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:36.398+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:36.420 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:36.420 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:36.429 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:36.499 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:36.499+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:36.600 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:36.600+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:36.603 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:36.603 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:36.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:36.700 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:36.700+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:36.752 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:36.753 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:36.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:36.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:29:36.807 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:36.805+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:36.897 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:36.897 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:36.902 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:36.902+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:36.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:37.002 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:37.002+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:37.073 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:37.073 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:37.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:37.094 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:37.094 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:37.129 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:37.128+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:37.194 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:37.205 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:37.204+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:37.232 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:37.233 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:37.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:37.298 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:37.298 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:37.305 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:37.305+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:37.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '13 of 30' 2024-06-03T05:29:37.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:29:37.400 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:37.400 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:37.403 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:37.407 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:37.407+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:37.461 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:37.462 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:37.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:37.500 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:37.501 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:37.544 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:37.543+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:37.608 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:37.608+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:37.626 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:37.626 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:37.646 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 319.97991171500001, 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:29:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:29:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 199.83193797499999, 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:29:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:29:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:37.655 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:37.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:37.709 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:37.708+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:37.745 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:37.745 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:37.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:37.793 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:37.793 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:37.793 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:37.794 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:37.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:37.847 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:37.847+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:37.911 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:37.911+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:37.937 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:37.937 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:37.937 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:37.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:38.031 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:38.031+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:38.113 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:38.113+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:38.173 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:38.173 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:38.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:38.173 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:38.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.174 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:29:38.174 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:38.174 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 339.30383784499998, 2024-06-03T05:29:38.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:29:38.174 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.174 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:29:38.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:29:38.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:29:38.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:29:38.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:29:38.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:29:38.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:29:38.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:29:38.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.181 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.181 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.181 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:38.181 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 338.12888151300001, 2024-06-03T05:29:38.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:29:38.181 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:38.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:38.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:29:38.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:29:38.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:29:38.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 338.12791907899998, 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:29:38.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:29:38.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:29:38.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 338.12712907600002, 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:29:38.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:38.198 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 338.09247521499998, 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:29:38.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.204 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.204 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:38.204 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 338.09178571799998, 2024-06-03T05:29:38.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:29:38.204 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.204 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.204 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.205 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:29:38.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.205 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:29:38.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:29:38.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 338.09111236899997, 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:38.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:29:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 338.01867925400001, 2024-06-03T05:29:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:29:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 337.26069206199998, 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:29:38.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:29:38.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 337.01204174100002, 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:38.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:29:38.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 337.01079635399998, 2024-06-03T05:29:38.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:29:38.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:29:38.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 337.00973684500002, 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:29:38.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:29:38.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 336.97688844999999, 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.242 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:29:38.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:29:38.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:38.247 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 336.97580331900002, 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:29:38.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 336.97478630099999, 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:29:38.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:29:38.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 336.95500278399999, 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:29:38.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:29:38.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:38.263 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 332.830394791, 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:29:38.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 332.829575127, 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.269 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:29:38.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:29:38.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 332.82887417799998, 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:38.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:29:38.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:29:38.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:29:38.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:38.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.279 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 326.42182044600003, 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.280 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:29:38.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:29:38.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.284 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.284 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.284 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:38.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:38.286 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:38.285+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:38.334 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:29:38.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '14 of 30' 2024-06-03T05:29:38.394 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:38.394+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:38.468 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:38.469 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:38.469 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:38.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:38.520 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:38.519+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:38.588 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:38.588+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:38.651 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:38.651 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:38.651 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:38.651 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:38.651 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:38.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:38.702 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:38.702+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:38.790 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:38.789+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:38.839 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:38.839 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:38.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:38.839 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 319.96798965599999, 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:29:38.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:29:38.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 199.85003051000001, 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:38.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:38.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:38.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:38.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:38.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:29:38.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:29:38.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.848 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:38.848 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:38.848 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:38.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:38.893 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:38.893+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:38.984 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:38.984 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:38.984 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:38.992 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:38.991+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:38.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:39.092 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:39.092+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:39.168 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:39.168 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:39.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:39.220 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:39.220+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:39.294 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:39.294+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:39.310 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:39.311 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:39.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:39.313 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:29:39.313 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:39.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:39.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '15 of 30' 2024-06-03T05:29:39.426 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:39.425+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:39.492 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:39.493 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:39.493 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:39.493 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:39.496 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:39.496+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:39.503 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:39.597 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:39.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:39.679 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:39.679 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:39.679 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:39.679 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:39.679 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:39.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:39.698 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:39.698+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:39.750 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:29:39.798 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:39.799+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:39.884 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:39.884 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:39.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:39.884 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:29:39.884 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:39.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:39.899 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:39.899+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:39.971 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:39.964+0000 5dce500 -1 osd.4 38 log_to_monitors true 2024-06-03T05:29:40.000 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:39.999+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:40.034 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:40.034 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:40.034 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:40.034 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:40.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:40.054 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:40.054 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:40.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:40.101 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:40.100+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:40.164 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:40.164 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:40.202 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:40.202+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:40.218 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:40.218 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:40.228 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:40.280 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:40.280 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:40.303 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:40.303+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:40.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:29:40.386 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:40.387 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:40.404 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:40.403+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:40.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '16 of 30' 2024-06-03T05:29:40.420 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:40.420 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:40.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:40.462 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:40.506 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:40.505+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:40.575 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:40.576 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:40.606 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:40.605+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:40.618 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:40.618 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:40.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:40.682 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:40.683 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:40.706 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:40.707+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:40.778 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:40.778 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:40.781 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:40.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:40.805 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:40.805 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:40.808 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:40.808+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:40.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:40.909 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:40.908+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:40.933 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:29:40.934 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:29:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:29:40.936 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-06-03T05:29:40.995 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:40.995 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:41.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:41.046 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:41.045+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:41.112 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:41.111+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:41.179 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:41.180 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:41.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:41.231 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:41.230+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:41.314 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:41.314+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:41.363 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:41.363 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:41.371 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:41.416 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:41.415+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:41.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '17 of 30' 2024-06-03T05:29:41.509 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:41.509 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:41.515 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:41.516+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:41.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:41.569 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:29:41.616 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:41.616+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:41.656 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:41.656 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:41.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:41.717 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:41.716+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:41.818 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:41.817+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:41.838 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:41.838 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:41.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:41.919 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:41.919+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:41.981 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:41.981 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:41.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:42.020 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:42.019+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:42.120 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:42.120+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:42.163 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:42.163 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:42.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:42.221 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:42.221+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:42.309 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:42.309 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:42.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:42.323 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:42.322+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:42.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '18 of 30' 2024-06-03T05:29:42.450 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:42.450+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:42.465 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:42.465 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:42.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:42.559 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:42.559+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:42.625 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:42.624+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:42.652 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:42.652 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:42.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:42.725 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:42.725+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:42.783 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:29:42.827 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:42.826+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:42.838 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:42.839 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:42.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:42.927 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:42.927+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:42.991 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:42.991 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:43.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:43.047 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:43.046+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:43.058 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:43.058 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:43.084 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:43.072+0000 14268640 -1 osd.4 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:29:43.129 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:43.129+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:43.149 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:43.149 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:43.149 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:43.149 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:43.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:43.164 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:43.164 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:43.229 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:43.230+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:43.260 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:43.261 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:43.330 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:43.330+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:43.331 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:43.331 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:43.331 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:43.331 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:43.331 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:43.342 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:43.388 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:43.388 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:43.391 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-03T05:29:43.432 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:43.431+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:43.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '19 of 30' 2024-06-03T05:29:43.490 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:43.490 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:43.532 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:43.532+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:43.557 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:43.557 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:43.558 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:29:43.558 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:43.568 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:43.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:29:43.588 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:43.589 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:43.633 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:43.633+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:43.718 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:43.719 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-03T05:29:43.723 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-03T05:29:43.733 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:43.733 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:43.733 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:43.733 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:43.733 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:43.733+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:43.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:43.834 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:43.834+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:43.919 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:43.919 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:43.919 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:43.919 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:43.919 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:43.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:43.970 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:43.969+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:44.036 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:44.036+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:44.137 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:44.137+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:44.148 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 332.32727309500001, 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:44.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:29:44.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:44.153 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 332.06844129199999, 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 167.996529337, 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:29:44.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:29:44.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 167.90638608500001, 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:44.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:44.165 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:44.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:44.240 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:44.238+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:44.309 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:44.309 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:44.309 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:44.310 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:44.319 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:44.339 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:44.339+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:44.440 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:44.440+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:44.497 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:44.497 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:44.498 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:44.498 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:44.498 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:44.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:44.527 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '20 of 30' 2024-06-03T05:29:44.568 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:44.567+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:44.612 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:02:01.254 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:44.613 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:02:01.255 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:44.633 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:02:01.275 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:44.643 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:44.642+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:44.652 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:44.652 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:44.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:44.652 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:44.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.652 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:44.652 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 168.690488168, 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:29:44.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:29:44.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 168.45511805800001, 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:29:44.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:29:44.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.660 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:44.661 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:44.661 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:44.662 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:44.670 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:57.949 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:44.674 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:57.952 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:44.697 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:00:57.976 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:29:44.744 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:44.744+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:44.765 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:29:44.858 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:44.855+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:44.871 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:44.871 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:44.871 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:44.871 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:44.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:44.946 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:44.946+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:45.023 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:45.023 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:45.023 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:45.023 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:45.023 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:45.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:45.075 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:45.074+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:45.148 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:45.148+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:45.236 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 327.566181814, 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:29:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 207.41820807400001, 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:29:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:29:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:45.245 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:45.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:45.249 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:45.248+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:45.350 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:45.349+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:45.388 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:45.388 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:45.388 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:45.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:45.451 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:45.451+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:45.534 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:45.534 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:45.534 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:45.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:45.584 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:45.584+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:45.654 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:45.653+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:45.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:29:45.768 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 346.89751074600002, 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:29:45.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.774 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 345.722554414, 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:29:45.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:29:45.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:29:45.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:29:45.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 345.72159198000003, 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.780 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:45.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:29:45.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:45.785 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 345.72080197700001, 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:29:45.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 345.68614811600003, 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.791 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:29:45.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:29:45.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 345.68545861899997, 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:45.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:29:45.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:29:45.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:45.801 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 345.68478527000002, 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:45.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.806 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 345.612352155, 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:29:45.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.811 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.85436496300002, 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.812 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:29:45.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:29:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.60571464200001, 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:29:45.817 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:29:45.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.60446925500003, 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:29:45.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:29:45.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.60340974600001, 2024-06-03T05:29:45.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.57056135099998, 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.834 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:29:45.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:29:45.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.56947622000001, 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:29:45.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.56845920199999, 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:29:45.844 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:29:45.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:45.849 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.54867568499998, 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:29:45.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:29:45.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.854 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 340.42406769199999, 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:29:45.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 340.42324802799999, 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.860 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:29:45.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:29:45.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 340.42254707900003, 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:29:45.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:29:45.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 334.01549334700002, 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:29:45.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:45.877 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:45.878 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:45.878 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:45.768+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:45.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:45.887 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:45.886+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:45.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:45.974 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:45.975 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:45.982 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:45.982+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:46.027 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:46.027 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:46.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:46.070 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:46.070 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:46.094 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:46.095 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:46.095 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:46.095+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:46.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-06-03T05:29:46.163 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:46.164 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:46.196 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:46.195+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:46.211 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:46.211 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:46.211 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:46.211 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:46.211 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:46.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:46.262 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:46.263 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:46.283 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:46.282+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:46.385 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:46.384+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:46.400 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:46.400 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 327.52619638300001, 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:46.401 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:29:46.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:29:46.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 207.40823723700001, 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:46.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:46.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:46.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:46.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:46.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:46.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:29:46.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:29:46.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:46.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:46.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:46.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:46.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:46.409 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:46.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:46.429 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:29:46.429 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-06-03T05:29:46.485 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:46.485+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:46.545 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:46.545 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:46.545 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:46.545 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:46.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:46.559 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:46.559 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:46.596 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:46.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:46.659 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:46.660 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:46.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '21 of 30' 2024-06-03T05:29:46.693 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:46.691+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:46.732 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:46.732 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:46.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:46.762 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:46.762 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:46.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:29:46.787 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:46.787+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:46.890 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:46.887+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:46.928 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:46.928 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:46.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:46.989 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:46.989+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:47.071 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:47.071 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:47.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:29:47.122 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:47.122+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:47.191 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:47.191+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:47.263 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:47.264 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:47.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:29:47.322 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:47.321+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:47.392 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:47.392+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:47.493 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:47.493+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:47.545 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:47.545 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:47.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:47.597 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:47.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:47.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '22 of 30' 2024-06-03T05:29:47.695 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:47.696+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:47.697 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:47.698 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:47.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:47.797 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:47.797+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:47.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:47.898 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:47.898+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:47.993 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:47.994 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:47.999 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:47.998+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:48.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:48.101 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:48.101+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:48.183 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:48.183 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:48.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:48.202 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:48.202+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:48.223 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-06-03T05:29:48.223 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:29:48.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-06-03T05:29:48.302 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:48.303+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:48.340 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:48.341 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:48.351 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:48.403 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:48.403+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:48.488 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:48.488 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:48.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:48.505 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:48.505+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:48.605 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:48.605+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:48.673 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:48.673 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:48.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:48.706 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:48.707+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:48.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:29:48.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '23 of 30' 2024-06-03T05:29:48.814 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:48.812+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:48.839 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:48.839 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:48.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:48.908 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:48.908+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:49.042 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:49.041+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:49.057 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:49.057 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:49.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:49.072 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:49.073 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:49.110 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:49.110+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:49.197 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:49.197 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:49.211 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:49.211+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:49.230 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:49.230 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:49.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:49.295 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:49.296 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:49.311 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:49.311+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:49.412 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:49.413+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:49.424 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:49.424 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:49.425 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:49.425 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:49.425 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:49.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:49.514 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:49.514+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:49.528 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:29:49.529 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:29:49.581 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:49.581 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 331.91133112199998, 2024-06-03T05:29:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:29:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:49.582 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:49.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:29:49.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 211.76335738200001, 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:49.587 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:49.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:29:49.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:49.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:49.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:29:49.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:29:49.591 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:49.591 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:49.591 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:49.591 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:49.591 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:49.591 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:49.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:49.594 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-06-03T05:29:49.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-06-03T05:29:49.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:49.635 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:49.638 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:49.638+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:49.716 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:49.716+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:49.732 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:49.732 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:49.732 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:49.732 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:49.735 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:49.735 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:49.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:49.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '24 of 30' 2024-06-03T05:29:49.826 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:49.826+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:49.848 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:49.848 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:49.917 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:49.917 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:49.917 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:49.917 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:49.917 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:49.918 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:49.917+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:49.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:50.018 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:50.018+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:50.120 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:50.120+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:50.151 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 351.28261990999999, 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:29:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:29:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 350.10766357799997, 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:29:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:29:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:29:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:29:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 350.106701144, 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:29:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:29:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:29:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 350.10591114099998, 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:29:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:29:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:29:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:29:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:29:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:29:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:29:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 350.07125728, 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:29:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:29:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 350.070567783, 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:29:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:29:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:29:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:29:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:29:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 350.06989443399999, 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:29:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:29:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:29:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:29:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:29:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 349.99746131900002, 2024-06-03T05:29:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:29:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:29:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:29:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:29:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:29:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:29:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:29:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 349.23947412699999, 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:29:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:29:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:29:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:29:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:29:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:29:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:29:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:29:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:29:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:29:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 348.99082380599998, 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:29:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:29:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:29:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:29:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:29:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:29:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:29:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 348.989578419, 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:29:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:29:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 348.98851890999998, 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:29:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:29:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 348.95567051500001, 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:29:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:29:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:29:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 348.95458538399998, 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.233 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:29:50.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 348.95356836600001, 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:29:50.238 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:29:50.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:29:50.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:29:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 348.93378484900001, 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:29:50.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.80917685600002, 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.80835719200002, 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:29:50.254 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:29:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.259 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 344.807656243, 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:29:50.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:29:50.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 338.40060251099999, 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:29:50.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.269 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:50.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:29:50.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:50.271 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:50.270+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:50.371 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:50.371+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:50.447 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:50.448 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:50.448 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:50.448 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:50.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:50.472 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:50.471+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:50.573 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:50.573+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:50.630 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:50.631 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:50.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:50.681 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:50.681+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:50.755 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.2 to 1 (10000) 2024-06-03T05:29:50.770 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-06-03T05:29:50.775 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:50.776+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:50.821 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:50.821 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:50.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:50.821 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:29:50.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.821 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:50.821 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:50.821 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 331.94926247199999, 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:29:50.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:29:50.826 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 211.83130332600001, 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:29:50.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:29:50.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:29:50.831 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:29:50.831 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:50.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:50.877 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:50.876+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:50.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '25 of 30' 2024-06-03T05:29:50.994 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:50.992+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:51.006 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:51.006 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:51.006 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:51.006 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:51.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:51.079 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:51.078+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:51.160 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:51.160 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:51.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:51.180 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:51.179+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:51.281 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:51.280+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:51.304 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:29:51.304 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:51.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:51.382 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:51.381+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:51.447 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:51.447 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:51.447 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:51.447 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:51.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:29:51.483 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:51.482+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:51.584 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:51.584+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:51.595 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:51.595 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:51.595 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:51.595 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:51.595 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:51.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:29:51.689 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:51.689+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:51.784 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:51.784 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:51.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:51.784 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:29:51.784 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:51.786 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:51.786+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:51.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:51.862 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:29:51.906 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:51.905+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:51.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '26 of 30' 2024-06-03T05:29:51.949 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:51.950 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:51.950 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:51.950 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:51.960 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:29:51.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:51.961 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-06-03T05:29:51.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-06-03T05:29:51.988 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:51.988+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:52.089 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:52.089+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:52.197 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:52.197 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:52.197 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:52.196+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:52.198 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:52.198 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:52.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:52.290 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:52.290+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:52.291 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:52.291 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:52.391 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:52.391+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:52.411 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:52.412 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:52.413 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:52.414 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:52.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:52.513 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:52.513+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:52.529 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:52.529 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:52.593 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:52.592+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:52.622 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:29:52.623 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:29:52.630 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:52.630 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:52.637 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:52.716 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:52.716 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:52.717 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:52.716+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:52.794 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:52.794+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:52.816 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:52.817 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:52.817 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:52.818 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:52.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:52.895 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:52.896+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:52.910 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:52.910 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:52.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '27 of 30' 2024-06-03T05:29:53.006 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:53.007 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:53.007 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:53.007+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:53.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.4 to 1 (10000) 2024-06-03T05:29:53.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-06-03T05:29:53.125 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:53.125+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:53.162 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:53.162 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:53.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:53.198 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:53.198+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:53.299 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:53.300+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:53.363 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:53.364 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:53.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:53.400 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:53.400+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:53.501 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:53.502+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:53.548 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:53.549 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:53.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:53.602 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:53.603+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:53.703 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:53.704+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:53.743 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:53.743 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:53.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:53.804 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:53.805+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:53.886 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:53.886 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:53.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:53.905 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:53.905+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:53.950 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '28 of 30' 2024-06-03T05:29:53.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:29:54.007 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:54.007+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:54.033 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:54.033 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:54.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:54.108 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:54.108+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:54.123 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.6 to 1 (10000) 2024-06-03T05:29:54.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-06-03T05:29:54.201 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:54.201 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:54.208 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:54.208+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:54.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:54.308 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:54.309+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:54.383 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:54.383 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:54.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:54.437 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:54.437+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:54.511 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:54.511+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:54.526 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:54.526 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:54.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:54.612 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:54.612+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:54.713 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:54.713 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:54.713 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:54.713+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:54.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:54.814 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:54.814+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:54.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log '29 of 30' 2024-06-03T05:29:54.897 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:54.897 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:54.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:54.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:29:54.939 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:54.938+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:55.016 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:55.016+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:55.040 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:55.040 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:55.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:55.117 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:55.116+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:55.193 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:55.193 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:55.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:55.221 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:55.221 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:55.244 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:55.243+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:55.319 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:55.319+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:55.362 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:55.362 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:55.376 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:55.376 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:55.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:29:55.420 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:55.420+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:55.479 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:55.479 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:55.512 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-06-03T05:29:55.512 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:29:55.521 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:55.521+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:55.527 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:55.527 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:55.529 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-06-03T05:29:55.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:29:55.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:29:55.622 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:55.622+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:55.634 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:55.634 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:55.687 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:55.688 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:55.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:29:55.743 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:55.742+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:55.756 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:29:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:29:55.852 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:55.852 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:55.852 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:29:55.852 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:29:55.852 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:55.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:29:55.867 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:55.867 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:55.867 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:55.866+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:55.924 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:55.923+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:55.964 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:56.010 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:56.010 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:29:56.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:29:56.010 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:29:56.010 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:56.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:29:56.063 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:56.063+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:56.066 INFO:tasks.mon_recovery:adding mon c back in 2024-06-03T05:29:56.066 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-03T05:29:56.066 DEBUG:teuthology.orchestra.run.smithi038:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-06-03T05:29:56.067 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:56.067 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:56.125 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:56.124+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:56.126 INFO:tasks.ceph.mon.c:Started 2024-06-03T05:29:56.126 INFO:tasks.mon_recovery.ceph_manager:waiting for quorum size 3 2024-06-03T05:29:56.127 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:29:56.203 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:56.203 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:56.203 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:56.203 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:56.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:29:56.226 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:56.226+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:56.249 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:29:56.249 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:29:56.264 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-06-03T05:29:55.508844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":204612,"kb_used_data":5444,"kb_used_omap":16,"kb_used_meta":190575,"kb_avail":660398268,"statfs":{"total":676457349120,"available":676247826432,"internally_reserved":0,"allocated":5574656,"data_stored":2909561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16767,"internal_metadata":195149441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":79,"apply_latency_ms":79,"commit_latency_ns":79000000,"apply_latency_ns":79000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.703096"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-03T05:29:49.681350+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:29:49.681350+0000","last_peered":"2024-06-03T05:29:49.681350+0000","last_clean":"2024-06-03T05:29:49.681350+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:29:49.681350+0000","last_undegraded":"2024-06-03T05:29:49.681350+0000","last_fullsized":"2024-06-03T05:29:49.681350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.158207+0000","last_change":"2024-06-03T05:29:45.112876+0000","last_active":"2024-06-03T05:29:52.158207+0000","last_peered":"2024-06-03T05:29:52.158207+0000","last_clean":"2024-06-03T05:29:52.158207+0000","last_became_active":"2024-06-03T05:29:45.083288+0000","last_became_peered":"2024-06-03T05:29:45.083288+0000","last_unstale":"2024-06-03T05:29:52.158207+0000","last_undegraded":"2024-06-03T05:29:52.158207+0000","last_fullsized":"2024-06-03T05:29:52.158207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:28:04.376061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":151,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-03T05:29:49.674991+0000","last_change":"2024-06-03T05:29:45.181097+0000","last_active":"2024-06-03T05:29:49.674991+0000","last_peered":"2024-06-03T05:29:49.674991+0000","last_clean":"2024-06-03T05:29:49.674991+0000","last_became_active":"2024-06-03T05:29:45.174231+0000","last_became_peered":"2024-06-03T05:29:45.174231+0000","last_unstale":"2024-06-03T05:29:49.674991+0000","last_undegraded":"2024-06-03T05:29:49.674991+0000","last_fullsized":"2024-06-03T05:29:49.674991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:02:58.418447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.046953+0000","last_change":"2024-06-03T05:28:55.653541+0000","last_active":"2024-06-03T05:29:52.046953+0000","last_peered":"2024-06-03T05:29:52.046953+0000","last_clean":"2024-06-03T05:29:52.046953+0000","last_became_active":"2024-06-03T05:28:55.625096+0000","last_became_peered":"2024-06-03T05:28:55.625096+0000","last_unstale":"2024-06-03T05:29:52.046953+0000","last_undegraded":"2024-06-03T05:29:52.046953+0000","last_fullsized":"2024-06-03T05:29:52.046953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:31:08.259117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":150,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-03T05:29:49.669151+0000","last_change":"2024-06-03T05:28:54.917780+0000","last_active":"2024-06-03T05:29:49.669151+0000","last_peered":"2024-06-03T05:29:49.669151+0000","last_clean":"2024-06-03T05:29:49.669151+0000","last_became_active":"2024-06-03T05:28:54.908410+0000","last_became_peered":"2024-06-03T05:28:54.908410+0000","last_unstale":"2024-06-03T05:29:49.669151+0000","last_undegraded":"2024-06-03T05:29:49.669151+0000","last_fullsized":"2024-06-03T05:29:49.669151+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:52:54.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.985068+0000","last_change":"2024-06-03T05:29:47.550761+0000","last_active":"2024-06-03T05:29:52.985068+0000","last_peered":"2024-06-03T05:29:52.985068+0000","last_clean":"2024-06-03T05:29:52.985068+0000","last_became_active":"2024-06-03T05:29:47.544310+0000","last_became_peered":"2024-06-03T05:29:47.544310+0000","last_unstale":"2024-06-03T05:29:52.985068+0000","last_undegraded":"2024-06-03T05:29:52.985068+0000","last_fullsized":"2024-06-03T05:29:52.985068+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:43:46.555272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-03T05:29:49.665844+0000","last_change":"2024-06-03T05:28:55.523772+0000","last_active":"2024-06-03T05:29:49.665844+0000","last_peered":"2024-06-03T05:29:49.665844+0000","last_clean":"2024-06-03T05:29:49.665844+0000","last_became_active":"2024-06-03T05:28:55.517462+0000","last_became_peered":"2024-06-03T05:28:55.517462+0000","last_unstale":"2024-06-03T05:29:49.665844+0000","last_undegraded":"2024-06-03T05:29:49.665844+0000","last_fullsized":"2024-06-03T05:29:49.665844+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T07:46:01.499697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-03T05:29:49.672855+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:29:49.672855+0000","last_peered":"2024-06-03T05:29:49.672855+0000","last_clean":"2024-06-03T05:29:49.672855+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:29:49.672855+0000","last_undegraded":"2024-06-03T05:29:49.672855+0000","last_fullsized":"2024-06-03T05:29:49.672855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":167,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.035141+0000","last_change":"2024-06-03T05:26:56.251989+0000","last_active":"2024-06-03T05:29:52.035141+0000","last_peered":"2024-06-03T05:29:52.035141+0000","last_clean":"2024-06-03T05:29:52.035141+0000","last_became_active":"2024-06-03T05:26:55.711401+0000","last_became_peered":"2024-06-03T05:26:55.711401+0000","last_unstale":"2024-06-03T05:29:52.035141+0000","last_undegraded":"2024-06-03T05:29:52.035141+0000","last_fullsized":"2024-06-03T05:29:52.035141+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:12:18.487760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378701,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":294615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411403,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":692224,"data_stored":307509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":4,"up_from":83,"seq":356482285571,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":27707,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":667648,"data_stored":294615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28372686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1085440,"data_stored":701051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411406,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1085440,"data_stored":701051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":692224,"data_stored":307509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":6,"up_from":71,"seq":304942678030,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":668,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":684032,"data_stored":303211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28503763},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":0,"seq":55834574922,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:29:56.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:29:56.327 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:56.326+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:56.358 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:56.358 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:56.358 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:56.358 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:56.358 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:56.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:29:56.428 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:56.428+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:56.505 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:56.505 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:56.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:56.505 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:29:56.505 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:56.514 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:29:56.557 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:56.557+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:56.631 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:56.630+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:57.094 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:57.094 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:57.094 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:57.094 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:57.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:29:57.097 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":32,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:29:57.098 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:57.095+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:57.103 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:29:57.103 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:29:57.117 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-06-03T05:29:56.362511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":204624,"kb_used_data":5460,"kb_used_omap":16,"kb_used_meta":190575,"kb_avail":660398256,"statfs":{"total":676457349120,"available":676247814144,"internally_reserved":0,"allocated":5591040,"data_stored":2918157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16767,"internal_metadata":195149441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":77,"apply_latency_ms":77,"commit_latency_ns":77000000,"apply_latency_ns":77000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.331814"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":148,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.561071+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:29:52.561071+0000","last_peered":"2024-06-03T05:29:52.561071+0000","last_clean":"2024-06-03T05:29:52.561071+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:29:52.561071+0000","last_undegraded":"2024-06-03T05:29:52.561071+0000","last_fullsized":"2024-06-03T05:29:52.561071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.158207+0000","last_change":"2024-06-03T05:29:45.112876+0000","last_active":"2024-06-03T05:29:52.158207+0000","last_peered":"2024-06-03T05:29:52.158207+0000","last_clean":"2024-06-03T05:29:52.158207+0000","last_became_active":"2024-06-03T05:29:45.083288+0000","last_became_peered":"2024-06-03T05:29:45.083288+0000","last_unstale":"2024-06-03T05:29:52.158207+0000","last_undegraded":"2024-06-03T05:29:52.158207+0000","last_fullsized":"2024-06-03T05:29:52.158207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:28:04.376061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":155,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.555847+0000","last_change":"2024-06-03T05:29:45.181097+0000","last_active":"2024-06-03T05:29:52.555847+0000","last_peered":"2024-06-03T05:29:52.555847+0000","last_clean":"2024-06-03T05:29:52.555847+0000","last_became_active":"2024-06-03T05:29:45.174231+0000","last_became_peered":"2024-06-03T05:29:45.174231+0000","last_unstale":"2024-06-03T05:29:52.555847+0000","last_undegraded":"2024-06-03T05:29:52.555847+0000","last_fullsized":"2024-06-03T05:29:52.555847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:02:58.418447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.046953+0000","last_change":"2024-06-03T05:28:55.653541+0000","last_active":"2024-06-03T05:29:52.046953+0000","last_peered":"2024-06-03T05:29:52.046953+0000","last_clean":"2024-06-03T05:29:52.046953+0000","last_became_active":"2024-06-03T05:28:55.625096+0000","last_became_peered":"2024-06-03T05:28:55.625096+0000","last_unstale":"2024-06-03T05:29:52.046953+0000","last_undegraded":"2024-06-03T05:29:52.046953+0000","last_fullsized":"2024-06-03T05:29:52.046953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:31:08.259117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.550061+0000","last_change":"2024-06-03T05:28:54.917780+0000","last_active":"2024-06-03T05:29:52.550061+0000","last_peered":"2024-06-03T05:29:52.550061+0000","last_clean":"2024-06-03T05:29:52.550061+0000","last_became_active":"2024-06-03T05:28:54.908410+0000","last_became_peered":"2024-06-03T05:28:54.908410+0000","last_unstale":"2024-06-03T05:29:52.550061+0000","last_undegraded":"2024-06-03T05:29:52.550061+0000","last_fullsized":"2024-06-03T05:29:52.550061+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:52:54.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.985068+0000","last_change":"2024-06-03T05:29:47.550761+0000","last_active":"2024-06-03T05:29:52.985068+0000","last_peered":"2024-06-03T05:29:52.985068+0000","last_clean":"2024-06-03T05:29:52.985068+0000","last_became_active":"2024-06-03T05:29:47.544310+0000","last_became_peered":"2024-06-03T05:29:47.544310+0000","last_unstale":"2024-06-03T05:29:52.985068+0000","last_undegraded":"2024-06-03T05:29:52.985068+0000","last_fullsized":"2024-06-03T05:29:52.985068+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:43:46.555272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":156,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.546816+0000","last_change":"2024-06-03T05:28:55.523772+0000","last_active":"2024-06-03T05:29:52.546816+0000","last_peered":"2024-06-03T05:29:52.546816+0000","last_clean":"2024-06-03T05:29:52.546816+0000","last_became_active":"2024-06-03T05:28:55.517462+0000","last_became_peered":"2024-06-03T05:28:55.517462+0000","last_unstale":"2024-06-03T05:29:52.546816+0000","last_undegraded":"2024-06-03T05:29:52.546816+0000","last_fullsized":"2024-06-03T05:29:52.546816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T07:46:01.499697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":148,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.553831+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:29:52.553831+0000","last_peered":"2024-06-03T05:29:52.553831+0000","last_clean":"2024-06-03T05:29:52.553831+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:29:52.553831+0000","last_undegraded":"2024-06-03T05:29:52.553831+0000","last_fullsized":"2024-06-03T05:29:52.553831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":167,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-06-03T05:29:52.035141+0000","last_change":"2024-06-03T05:26:56.251989+0000","last_active":"2024-06-03T05:29:52.035141+0000","last_peered":"2024-06-03T05:29:52.035141+0000","last_clean":"2024-06-03T05:29:52.035141+0000","last_became_active":"2024-06-03T05:26:55.711401+0000","last_became_peered":"2024-06-03T05:26:55.711401+0000","last_unstale":"2024-06-03T05:29:52.035141+0000","last_undegraded":"2024-06-03T05:29:52.035141+0000","last_fullsized":"2024-06-03T05:29:52.035141+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:12:18.487760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":795680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378702,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":303211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411403,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":692224,"data_stored":307509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":4,"up_from":83,"seq":356482285571,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":27707,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":667648,"data_stored":294615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28372686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1085440,"data_stored":701051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411406,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1085440,"data_stored":701051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476811,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":692224,"data_stored":307509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":6,"up_from":71,"seq":304942678030,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32748,"kb_used_data":668,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94339092,"statfs":{"total":96636764160,"available":96603230208,"internally_reserved":0,"allocated":684032,"data_stored":303211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28503763},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":0,"seq":55834574922,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:29:57.118 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-06-03T05:29:57.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:29:57.194 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:57.195+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:57.238 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:57.238 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:57.238 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:57.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:29:57.332 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:29:57.332+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:29:57.396 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:57.396+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:57.435 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 345.61270274999998, 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:29:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 345.35387094700002, 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:29:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:29:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 181.281958992, 2024-06-03T05:29:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:29:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 181.19181574000001, 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:29:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:57.452 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:57.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:29:57.497 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:57.497+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:57.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:29:57.598 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:57.597+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:57.632 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:57.632 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:57.632 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:57.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:29:57.699 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:57.699+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:57.783 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:57.784 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:57.784 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:57.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:29:57.803 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:57.802+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:57.901 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:57.900+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:57.991 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 182.025913323, 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:57.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:29:57.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:57.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 181.79054321300001, 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:29:57.996 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:57.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:57.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:29:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:29:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:58.000 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:58.000 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:58.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:29:58.002 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:58.001+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:58.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:29:58.130 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:58.130+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:58.161 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:58.161 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:29:58.161 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:58.170 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:29:58.203 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:58.203+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:58.304 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:58.305+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:58.368 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:58.369 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:29:58.369 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:29:58.369 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:29:58.369 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:58.380 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:29:58.390 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:29:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:29:58.434 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:58.434+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:58.495 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:29:58.496 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:29:58.505 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:58.506+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:58.553 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:29:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:29:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:29:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:29:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:29:58.553 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 340.88340853900002, 2024-06-03T05:29:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:29:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:58.554 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:58.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:58.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 220.73543479899999, 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:29:58.561 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:29:58.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:29:58.565 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:29:58.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:29:58.594 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:29:58.594 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:29:58.623 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:58.623+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:58.688 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:29:58.689 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:29:58.700 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:29:58.700 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:29:58.700 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:29:58.700 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:29:58.707 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:58.706+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:58.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:29:58.788 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:29:58.788 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:29:58.809 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:58.808+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:58.881 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:29:58.881 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:29:58.910 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:58.910+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:58.920 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:58.920 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:58.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:29:58.977 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:29:58.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:29:59.014 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:29:59.013+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:29:59.078 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:29:59.078 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:29:59.111 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:59.111 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:59.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:29:59.132 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:59.132+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:59.213 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:29:59.212+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:29:59.288 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:29:59.299 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:59.299 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:59.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:29:59.349 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:59.348+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:59.415 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:29:59.415+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:29:59.480 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:59.481 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:59.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:29:59.535 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:29:59.534+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:29:59.618 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:59.618+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:59.625 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:59.625 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:59.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:29:59.719 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:29:59.719+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:29:59.808 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:59.808 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:59.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:29:59.820 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:29:59.820+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:29:59.920 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:29:59.920+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:29:59.949 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:29:59.949 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:29:59.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:30:00.021 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:00.022+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:00.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:30:00.137 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:00.137 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:00.143 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:00.143+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:00.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:30:00.224 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:00.223+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:00.282 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:00.282 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:00.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:30:00.333 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:00.332+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:00.426 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:00.425+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:00.464 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:00.464 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:00.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:30:00.558 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:00.557+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:00.628 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:00.627+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:00.646 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:00.646 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:00.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:30:00.729 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:00.729+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:00.814 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:00.815 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:00.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:30:00.829 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:00.828+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:00.830 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":36,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:30:00.930 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:00.930+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:00.970 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:00.970 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:00.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:30:01.031 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:01.031+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:01.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:30:01.132 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:01.132+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:01.157 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:01.157 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:01.167 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:30:01.233 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:01.233+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:01.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:30:01.309 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:01.310 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:01.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:30:01.334 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:01.333+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:01.419 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:30:01.420 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:30:01.434 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:01.434+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:01.546 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:30:01.547 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:30:01.547 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:01.547+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:01.548 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:01.548 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:01.548 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:01.548 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:01.548 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:01.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:30:01.636 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:01.635+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:01.644 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:30:01.644 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:30:01.736 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:01.736+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:01.739 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:30:01.740 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:30:01.744 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:01.744 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:01.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:01.744 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:30:01.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.744 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:01.744 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:30:01.744 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 349.92271850700001, 2024-06-03T05:30:01.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:01.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:01.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:30:01.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:01.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 349.66388670399999, 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:30:01.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:01.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:30:01.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:30:01.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:01.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:30:01.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:30:01.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:30:01.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 185.591974749, 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:30:01.753 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:01.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:01.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 185.50183149700001, 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:01.757 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:30:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:01.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:30:01.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:30:01.761 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:01.761 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:01.761 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:01.761 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:01.761 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:01.761 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:01.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:30:01.835 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:30:01.836 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:30:01.838 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:01.837+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:01.907 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:01.907 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:01.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:30:01.947 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:30:01.947 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:30:01.948 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:01.946+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:02.038 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:02.039+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:02.056 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:30:02.056 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:30:02.063 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:02.063 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:02.063 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:02.063 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:02.063 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:02.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:30:02.119 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-06-03T05:30:02.119 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:30:02.139 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:02.138+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:02.148 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:30:02.148 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:30:02.241 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:02.240+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 186.29316464300001, 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:02.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:02.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:30:02.259 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 186.05779453299999, 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:02.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:30:02.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:02.264 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:02.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:30:02.342 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:02.342+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:02.420 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:02.420 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:02.420 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:02.421 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:02.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:30:02.442 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:02.441+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:02.463 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:30:02.544 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:02.544+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:02.598 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:02.599 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:02.599 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:02.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:30:02.645 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:02.644+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:02.746 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:02.746+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:02.789 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:02.789 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 345.12300516599998, 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:30:02.790 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:30:02.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.794 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 224.97503142599999, 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:30:02.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:02.798 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:02.799 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:02.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:30:02.847 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:02.846+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:02.934 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:02.934 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:02.935 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:02.935 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:02.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:30:02.987 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:02.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:03.048 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:03.049+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:03.145 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:03.145 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:03.145 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:03.149 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:03.148+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:03.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:30:03.250 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:03.249+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:03.351 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:03.351+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:03.379 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:03.379 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 364.51020290100001, 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.380 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:30:03.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:30:03.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.385 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 363.33524656899999, 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:30:03.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:30:03.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 363.33428413500002, 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:30:03.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:30:03.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 363.333494132, 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:30:03.396 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:30:03.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:30:03.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 363.29884027100002, 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.402 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:30:03.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:30:03.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:30:03.407 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 363.29815077400002, 2024-06-03T05:30:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:30:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:30:03.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:30:03.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 363.29747742500001, 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:30:03.413 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:30:03.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.418 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 363.22504430999999, 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:30:03.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 362.46705711800001, 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.424 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:30:03.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:30:03.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 362.218406797, 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:30:03.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.434 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 362.21716141000002, 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 362.216101901, 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.440 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:30:03.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 362.18325350600003, 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.445 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:30:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:30:03.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 362.182168375, 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:30:03.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:30:03.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 362.18115135699998, 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:30:03.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:30:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:30:03.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 362.16136784000003, 2024-06-03T05:30:03.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:30:03.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:30:03.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:30:03.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:30:03.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 358.03675984699998, 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:30:03.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 358.03594018299998, 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:30:03.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:30:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:30:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 358.03523923400002, 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:30:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:30:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:30:03.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 351.62818550200001, 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:30:03.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:03.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:03.487 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:03.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:30:03.489 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:03.489+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:03.590 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:03.589+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:03.663 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:03.663 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:03.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:30:03.690 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:03.691+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:03.792 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:03.791+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:03.845 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:30:03.846 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:03.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:03.846 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:03.846 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:03.846 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:03.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:30:03.893 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:03.893+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:03.993 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:03.994+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:04.033 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 345.16178566600001, 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:04.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:04.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:30:04.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 225.04382652000001, 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:30:04.038 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:04.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:04.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:04.043 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:04.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:30:04.094 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:04.094+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:04.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:30:04.178 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:04.178 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:04.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:30:04.201 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:08.012 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:04.202 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:04.202+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:04.296 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:04.296+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:04.329 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:04.329 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:04.329 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:04.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:30:04.398 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:04.398+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:04.441 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:30:04.442 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:30:04.471 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:04.472 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:04.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:04.472 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:30:04.472 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:04.477 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:08.288 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:04.479 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:30:04.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:30:04.525 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:04.525+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:04.569 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":40,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:30:04.590 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:30:04.590 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:30:04.599 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:04.599+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:04.616 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:04.616 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:04.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:30:04.709 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:04.709+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:04.712 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:30:04.712 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:30:04.799 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:04.799 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:04.799 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:04.801 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:04.801+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:04.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:30:04.823 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:30:04.824 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:30:04.902 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:04.902+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:04.928 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:30:04.928 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:30:04.992 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:04.992 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:05.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:30:05.003 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:05.003+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:05.052 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:30:05.052 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:30:05.104 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:05.104+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:05.154 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:05.155 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:05.161 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:30:05.162 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:30:05.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:30:05.207 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:05.206+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:05.263 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:30:05.263 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:30:05.305 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:05.306+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:05.343 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:05.343 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:05.352 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:30:05.407 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:05.407+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:05.486 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:05.486 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:05.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:30:05.539 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:05.539+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:05.609 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:05.609+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:05.700 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:05.701 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:05.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:30:05.711 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:05.710+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:05.811 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:05.812+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:05.857 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:05.858 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:05.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:30:05.952 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:05.952+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:06.014 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:06.014+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:06.043 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:06.043 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:06.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:30:06.115 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:06.115+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:06.124 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:30:06.216 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:06.216+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:06.235 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:06.236 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:06.245 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:30:06.330 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:06.330+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:06.418 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:06.418+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:06.422 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:06.422 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:06.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:30:06.519 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:06.519+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:06.619 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:06.620+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:06.620 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:06.620 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:06.632 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:30:06.720 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:06.720+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:06.806 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:06.806 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:06.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:30:06.860 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:06.860+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:06.923 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:06.923+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:06.966 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:06.966 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:06.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:30:07.023 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:07.024+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:07.120 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-06-03T05:30:07.120 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:30:07.121 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-06-03T05:30:07.121 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-03T05:30:07.121 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-03T05:30:07.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-06-03T05:30:07.124 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:07.125+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:07.154 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:07.155 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:07.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:30:07.225 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:07.225+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:07.269 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:07.269 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "" 2024-06-03T05:30:07.270 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:07.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:30:07.306 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:07.306 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:07.315 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:30:07.325 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:07.326+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:07.427 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:07.427+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:07.491 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:07.491 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:07.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:30:07.527 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:07.527+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:07.538 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:30:07.539 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:30:07.588 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:30:07.632 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:30:07.633 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:30:07.633 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:07.633+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:07.673 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:07.673 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:07.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:30:07.729 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:07.729+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:07.738 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:30:07.738 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:30:07.820 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:07.820 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:07.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:30:07.833 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:30:07.833 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:30:07.876 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:07.876+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:07.931 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:07.932+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:07.938 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:30:07.938 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:30:08.005 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:08.005 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:08.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:30:08.032 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:30:08.033 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:30:08.033 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:08.033+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:08.134 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:08.134+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:30:08.136 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:30:08.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:30:08.244 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:08.243+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:08.244 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:30:08.244 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:30:08.252 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:30:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:30:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:08.252 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 349.37992715299998, 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:08.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.256 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 229.26196800700001, 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.257 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:30:08.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:08.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:08.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:08.261 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:08.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:30:08.306 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":43,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:30:08.347 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:08.347+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:08.436 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:08.436 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:08.436 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:08.436 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:08.437 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:08.437+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:08.445 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:30:08.538 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:08.538+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:08.619 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:08.619 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:08.619 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:08.619 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:08.619 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:08.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:30:08.673 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:08.673+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:08.748 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:08.748+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:08.764 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:08.764 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:08.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:08.764 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:30:08.764 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:08.774 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:30:08.841 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:08.841+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:08.945 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:08.944+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:08.950 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:08.950 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:08.950 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:08.950 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:08.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:30:09.045 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:09.045+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:09.135 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:09.135 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:09.135 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:09.144 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:09.144+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:09.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:30:09.245 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:09.245+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:09.335 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:09.335 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:09.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:09.335 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:30:09.335 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:09.344 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:30:09.346 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:09.346+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:09.406 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:30:09.447 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:09.447+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:09.481 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:09.481 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:09.481 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:09.481 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:09.490 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:30:09.547 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:09.547+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:09.622 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:09.623 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:09.623 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:09.623 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:09.623 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:09.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:30:09.674 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:09.673+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:09.748 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:09.749+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:09.809 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:30:09.810 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:09.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:30:09.849 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:09.849+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:09.951 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:09.951+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:09.995 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:09.995 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:09.995 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:09.995 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:10.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:30:10.052 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:10.052+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:10.138 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:10.139 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:10.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:30:10.152 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:10.152+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:10.253 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:10.254+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:10.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 358.46882682900002, 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:30:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:30:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 358.209995026, 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:30:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 194.13808307100001, 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:30:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:30:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 194.04793981899999, 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:30:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:10.306 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:10.307 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:30:10.354 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:10.354+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:10.455 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:10.455+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:10.485 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:10.485 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:10.485 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:10.485 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:10.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:30:10.555 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:30:10.556 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:30:10.556 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:10.556+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:10.657 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:10.657+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:10.666 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:30:10.666 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:30:10.703 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:10.704 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:10.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:30:10.773 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:30:10.773 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:30:10.775 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:10.775+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:10.858 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:10.858+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:10.877 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:30:10.877 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:30:10.891 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:10.891 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:10.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:30:10.971 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:30:10.971 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:30:10.971 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:10.972+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:11.063 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:30:11.064 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:30:11.064 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:11.063+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:11.075 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:11.075 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:11.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:30:11.159 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:30:11.159 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:30:11.161 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:11.162+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:11.251 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:30:11.252 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:30:11.261 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:11.261 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:11.262 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:11.262+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:11.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:30:11.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:30:11.399 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:11.399+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:11.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-06-03T05:30:11.453 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:11.453 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:11.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:30:11.464 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:11.464+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:11.565 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:11.565+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:11.604 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:11.604 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:11.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:30:11.665 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:11.666+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:11.766 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:11.767 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:11.767 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:11.767+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:11.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:30:11.867 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:11.868+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:11.913 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:11.913 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:11.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:30:11.969 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:11.969+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:12.057 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:12.057 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:12.067 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:30:12.070 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:12.070+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":47,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:30:12.170 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:12.171+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:12.240 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:12.240 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:12.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:30:12.279 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-06-03T05:30:12.279 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:30:12.280 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-06-03T05:30:12.280 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-03T05:30:12.280 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-03T05:30:12.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-06-03T05:30:12.294 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:12.294+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:12.372 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:12.372+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:12.384 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:12.384 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:12.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "" 2024-06-03T05:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:12.479 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:12.479+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:12.571 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:12.571 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:12.573 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:12.574+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:12.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:30:12.613 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-06-03T05:30:12.675 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:12.675+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:12.753 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:12.753 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:12.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:30:12.807 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:12.807+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:12.876 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:12.876+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:12.896 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:12.897 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:12.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:30:12.990 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:12.990+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:13.078 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:13.078+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:13.082 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:13.083 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:13.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:30:13.179 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:13.179+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:13.264 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:13.264 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:13.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:30:13.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:30:13.317 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:13.317+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:13.381 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:13.381+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:13.410 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:13.410 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:13.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:30:13.542 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:30:13.542 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:13.541+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:13.581 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:13.581 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:13.581 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:13.581 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:13.582 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:13.582+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:13.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:30:13.643 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:30:13.643 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:30:13.683 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:13.683+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:13.724 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:13.724 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:13.724 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:13.724 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:13.724 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:13.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:30:13.738 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:30:13.738 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:30:13.784 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:13.784+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:13.834 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:30:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:30:13.867 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:13.867 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:13.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:13.867 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-06-03T05:30:13.867 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:13.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:30:13.885 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:13.885+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:13.931 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:30:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:30:13.986 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:13.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:14.029 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:30:14.029 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:30:14.057 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:14.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:14.057 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:14.057 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:14.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:30:14.108 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:14.108+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:30:14.128 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:30:14.188 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:14.188+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:14.220 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:30:14.220 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:30:14.242 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:14.242 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:14.242 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:14.242 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:14.243 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:14.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:30:14.289 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:14.289+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:14.390 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:14.390+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:14.445 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:14.445 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:30:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:30:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 362.62366747300001, 2024-06-03T05:30:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18933038699999999, 2024-06-03T05:30:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:14.446 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:30:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813449+0000", 2024-06-03T05:30:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:14.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.813460+0000", 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.841003+0000", 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027543358 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.843959+0000", 2024-06-03T05:30:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029553019999999999 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.853382+0000", 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094229380000000005 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.865415+0000", 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012033624999999999 2024-06-03T05:30:14.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:11.870958+0000", 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055421519999999998 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.002779+0000", 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13182176200000001 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:14.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 20/18 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:30:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 362.36483566999999, 2024-06-03T05:30:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029320188, 2024-06-03T05:30:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072281+0000", 2024-06-03T05:30:14.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.072297+0000", 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6280999999999999e-05 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077329+0000", 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050316500000000004 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:14.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.077704+0000", 2024-06-03T05:30:14.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037510700000000001 2024-06-03T05:30:14.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:14.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.078148+0000", 2024-06-03T05:30:14.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044435500000000001 2024-06-03T05:30:14.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:14.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080473+0000", 2024-06-03T05:30:14.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023251270000000002 2024-06-03T05:30:14.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:14.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.080800+0000", 2024-06-03T05:30:14.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032720500000000001 2024-06-03T05:30:14.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:14.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:12.101601+0000", 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020800463000000002 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)])", 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 198.292923715, 2024-06-03T05:30:14.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027352887999999999, 2024-06-03T05:30:14.459 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:14.459 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:14.459 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:14.459 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:14.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:14.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144193+0000", 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.144204+0000", 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.168e-05 2024-06-03T05:30:14.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.148281+0000", 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040762480000000002 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149455+0000", 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011741049999999999 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:14.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.149707+0000", 2024-06-03T05:30:14.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025228099999999998 2024-06-03T05:30:14.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:14.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.151157+0000", 2024-06-03T05:30:14.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014497679999999999 2024-06-03T05:30:14.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:14.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.171546+0000", 2024-06-03T05:30:14.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020388805999999999 2024-06-03T05:30:14.463 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.463 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:14.463 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(1.0 46/44 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 198.20278046300001, 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012963552, 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:30:14.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:14.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234336+0000", 2024-06-03T05:30:14.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:14.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234346+0000", 2024-06-03T05:30:14.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0173000000000001e-05 2024-06-03T05:30:14.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:14.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234448+0000", 2024-06-03T05:30:14.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102391 2024-06-03T05:30:14.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:14.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234610+0000", 2024-06-03T05:30:14.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161201 2024-06-03T05:30:14.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:14.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.234781+0000", 2024-06-03T05:30:14.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171798 2024-06-03T05:30:14.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.242217+0000", 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074354269999999997 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.247299+0000", 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050825619999999997 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.469 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:14.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.470 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:14.470 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:14.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:30:14.491 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:14.491+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:14.592 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:14.592+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:14.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-06-03T05:30:14.647 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:14.647 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:14.647 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:14.647 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:14.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:30:14.699 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:14.698+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:14.794 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:14.794+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:14.845 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:14.845 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:14.846 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:14.846 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:14.846 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:14.854 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:30:14.897 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:14.894+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:14.992 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:14.992 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:14.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:14.992 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 17'75, data_included: [0~4096,65536~66064], data_size: 70160, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@17'75, size: 131600, copy_subset: [(0, 131600)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 131600, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 199.03105900099999, 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16363466900000001, 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:30:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957428+0000", 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.957445+0000", 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6249e-05 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:14.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.984311+0000", 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026866528000000001 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:55.992065+0000", 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007753383 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.001380+0000", 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093154239999999992 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.032399+0000", 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03101932 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.040598+0000", 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081986850000000007 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.121063+0000", 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.080465079999999994 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:14.996 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 46/44 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 17'76, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@17'76, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 198.795688891, 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024654110999999999, 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:30:14.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192799+0000", 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.192814+0000", 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4924e-05 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:14.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199332+0000", 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065185699999999996 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.199922+0000", 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000590308 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.200682+0000", 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075947700000000005 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:14.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.203768+0000", 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030860710000000001 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.204072+0000", 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030402700000000001 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:56.217453+0000", 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013380734 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:15.000 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:15.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:15.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:15.001 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:15.001 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:15.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:30:15.003 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:15.002+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:15.107 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:15.105+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:15.122 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:30:15.139 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:15.139 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:15.139 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:15.139 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:15.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:30:15.236 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:15.236+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:15.305 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:15.305+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:15.333 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:15.333 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:15.333 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:15.333 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:15.333 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:15.343 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:30:15.428 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:15.428+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:15.507 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:15.507+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:15.523 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:15.523 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:15.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 357.856986457, 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:30:15.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:30:15.527 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.70901271700001, 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:30:15.528 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:30:15.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:30:15.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:30:15.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:15.532 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:15.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:15.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:15.532 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:15.532 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:15.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:30:15.608 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:15.608+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:15.667 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:15.667 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:15.667 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:15.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:30:15.720 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:15.720+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:15.810 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:15.810+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:15.810 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:15.810 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:15.811 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:15.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:30:15.840 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":51,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:30:15.848 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-06-03T05:30:15.911 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:15.911+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:16.012 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:16.012+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:16.050 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:16.050 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:16.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:16.050 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 377.18125214100002, 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:30:16.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:30:16.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 376.00629580899999, 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:30:16.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 376.00533337500002, 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:30:16.062 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:30:16.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 376.004543372, 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:30:16.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:30:16.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 375.96988951100002, 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:30:16.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:30:16.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:30:16.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 375.96920001400002, 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:30:16.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 375.96852666500001, 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:30:16.083 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:30:16.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.088 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 375.89609354999999, 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:30:16.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:30:16.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:30:16.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:30:16.094 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 375.13810635800002, 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:30:16.099 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 374.889456037, 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.100 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:30:16.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:30:16.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:30:16.105 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 374.88821065000002, 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:30:16.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:30:16.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:30:16.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 374.887151141, 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.111 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:30:16.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 374.85430274599997, 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.116 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:30:16.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:30:16.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.121 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 374.85321761500001, 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:30:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:30:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:30:16.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:30:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:30:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 374.85220059699998, 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.129 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:30:16.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:16.134 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:30:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 374.83241708000003, 2024-06-03T05:30:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:30:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:30:16.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 370.70780908699999, 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:30:16.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:30:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:30:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:30:16.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:30:16.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 370.70698942299998, 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:30:16.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 370.70628847400002, 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:30:16.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:30:16.157 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 364.29923474200001, 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:30:16.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:30:16.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.161 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:16.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:30:16.163 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:16.163+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:16.233 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:30:16.277 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:16.277+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:16.296 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:16.297 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:16.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:30:16.365 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:16.365+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:16.450 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:16.450 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:16.450 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:16.450 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:16.451 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:16.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:30:16.466 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:16.466+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:16.491 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:30:16.491 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:30:16.567 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:16.567+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:16.586 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:30:16.586 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:30:16.599 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:16.599 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 357.72820580400003, 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:16.600 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:30:16.604 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 237.61024665799999, 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:30:16.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:16.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:16.609 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:16.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:30:16.680 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:30:16.694 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:16.694+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:16.769 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:16.769+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:16.776 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:30:16.777 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:30:16.786 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:16.787 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:16.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:30:16.877 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:30:16.877 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:30:16.881 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:16.881+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:16.970 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:16.970+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:16.982 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:30:16.983 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:30:17.015 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:17.016 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:17.016 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:17.016 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:17.016 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:17.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:30:17.070 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:17.071+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:17.087 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:30:17.087 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:30:17.172 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:17.172+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:30:17.184 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:30:17.201 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:17.201 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:17.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:30:17.272 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:17.273+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:17.343 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:17.343 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:17.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:30:17.397 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:17.397+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:17.424 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-06-03T05:30:17.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:30:17.475 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:17.475+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:17.488 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:17.489 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:17.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:30:17.583 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:17.583+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:17.672 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:17.672 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:17.676 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:17.676+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:17.681 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:30:17.778 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:17.778+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:17.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:30:17.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:30:17.879 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:17.878+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:17.991 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:17.991+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:18.007 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:18.007 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:18.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:30:18.101 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:18.101+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:18.182 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:18.182+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:18.191 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:18.191 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:18.201 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:30:18.285 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:18.285+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:18.377 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:18.377 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:18.384 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:18.384+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:18.387 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:30:18.485 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:18.485+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:18.569 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:18.569 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:18.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:30:18.586 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:18.586+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:18.687 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:18.687+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:18.711 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:18.711 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:18.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:30:18.805 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:18.805+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:18.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:30:18.897 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:18.897 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:18.898 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:18.898+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:18.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:30:18.991 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:18.991+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:19.082 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:19.082 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:19.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:30:19.092 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:19.092+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:19.192 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:19.193+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:19.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-03T05:30:19.288 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:19.288 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:19.293 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:19.294+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:19.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:30:19.394 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:19.395+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:19.485 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:30:19.485 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:30:19.490 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:19.490 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:19.495 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:19.495+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:19.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:30:19.595 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:19.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:19.599 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:30:19.601 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:30:19.614 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":55,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:30:19.669 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:30:19.679 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:19.680 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:19.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:30:19.696 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:19.696+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:19.696 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:30:19.697 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:30:19.797 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:19.797+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:19.805 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:30:19.805 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:30:19.847 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 362.18108315299997, 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.122251495, 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:30:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663144+0000", 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:19.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.663349+0000", 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204997 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.669145+0000", 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057968000000000004 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.673755+0000", 2024-06-03T05:30:19.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046097960000000002 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.683572+0000", 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098162330000000006 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.709425+0000", 2024-06-03T05:30:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025853477999999999 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:24:17.785395+0000", 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075970191000000006 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4164.0:5 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:30:19.852 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 242.03310941300001, 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077840690000000002, 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4164", 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:30:19.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811117+0000", 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811138+0000", 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0577e-05 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811343+0000", 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020512199999999999 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811579+0000", 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023583400000000001 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.811798+0000", 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219175 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.817128+0000", 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053300559999999997 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-06-03T05:26:17.818901+0000", 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001773305 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-06-03T05:30:19.856 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-06-03T05:30:19.857 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:19.857 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:30:19.898 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:30:19.898 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:30:19.902 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:19.902+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:19.996 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:30:19.996 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:30:20.000 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:20.000+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:20.038 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:20.039 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:20.039 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:20.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-06-03T05:30:20.098 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:30:20.098 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:30:20.133 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:20.133+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:20.189 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:30:20.190 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:30:20.201 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:30:20.202+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:30:20.222 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:20.222 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:20.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-06-03T05:30:20.316 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:20.316+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:20.403 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:20.403+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:20.457 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:30:20.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 381.58774708099997, 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38793674500000003, 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-06-03T05:30:20.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819107+0000", 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.819768+0000", 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066023099999999995 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.825787+0000", 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060197209999999996 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.830366+0000", 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045789589999999996 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.839604+0000", 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092377610000000006 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.856102+0000", 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016497837000000001 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:58.965586+0000", 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109484057 2024-06-03T05:30:20.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.173359+0000", 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20777282899999999 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.184126+0000", 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010767123 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.207044+0000", 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022918226999999999 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:30 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 380.41279074900001, 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18912958799999999, 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 30 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994064+0000", 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994076+0000", 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2033e-05 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994350+0000", 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027445200000000001 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994514+0000", 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016402100000000001 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.994686+0000", 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172015 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.996906+0000", 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002219562 2024-06-03T05:30:20.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.999647+0000", 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027410170000000001 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.178360+0000", 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17871347200000001 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.180345+0000", 2024-06-03T05:30:20.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001985068 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183193+0000", 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028479479999999999 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:31 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 380.41182831499998, 2024-06-03T05:30:20.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19270567499999999, 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 31 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995026+0000", 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995036+0000", 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0061000000000001e-05 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995173+0000", 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013638999999999999 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995325+0000", 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152251 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995488+0000", 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000163301 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.011199+0000", 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015710432999999999 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.013626+0000", 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024270279999999999 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.183437+0000", 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16981101800000001 2024-06-03T05:30:20.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.185014+0000", 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015778120000000001 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187732+0000", 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002717381 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.473 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:32 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 380.41103831200002, 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19645648099999999, 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 32 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995816+0000", 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:23:59.995836+0000", 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9992000000000001e-05 2024-06-03T05:30:20.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029736+0000", 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033900092999999999 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.029913+0000", 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017631100000000001 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030170+0000", 2024-06-03T05:30:20.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025705900000000002 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.032598+0000", 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024279380000000001 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.034374+0000", 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001776646 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.187967+0000", 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.153592438 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.189542+0000", 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015749010000000001 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.192273+0000", 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027311029999999999 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.478 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:33 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 380.37638445099998, 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16950174000000001, 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.479 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030470+0000", 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030483+0000", 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2860999999999999e-05 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030623+0000", 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140424 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030765+0000", 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014165 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.030921+0000", 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015569 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.050039+0000", 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019117891000000001 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.052234+0000", 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021950670000000002 2024-06-03T05:30:20.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.196447+0000", 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14421328899999999 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.198049+0000", 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001601924 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.199972+0000", 2024-06-03T05:30:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001922944 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 380.37569495399998, 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17216516800000001, 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.484 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031160+0000", 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031170+0000", 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0081000000000001e-05 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031304+0000", 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013460499999999999 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031444+0000", 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140273 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031592+0000", 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014755500000000001 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.062428+0000", 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030836039999999999 2024-06-03T05:30:20.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.064179+0000", 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017509750000000001 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.200219+0000", 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13604022700000001 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.201399+0000", 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001180022 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203325+0000", 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019253899999999999 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:30:20.489 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 380.37502160499997, 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174473726, 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:30:20.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031833+0000", 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.031843+0000", 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0009e-05 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089690+0000", 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057847141999999997 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.089879+0000", 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188581 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.090047+0000", 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168188 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.106699+0000", 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652377999999999 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.109642+0000", 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425200000000001 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.203560+0000", 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093918644999999995 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.204622+0000", 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001061324 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206307+0000", 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016849390000000001 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 380.30258849000001, 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10475707300000001, 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104266+0000", 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104280+0000", 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3601000000000001e-05 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104446+0000", 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000166572 2024-06-03T05:30:20.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104824+0000", 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037743299999999999 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.104999+0000", 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174909 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.127153+0000", 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022154876 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.129586+0000", 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024327440000000001 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.206453+0000", 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076867014999999997 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.207394+0000", 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094068099999999996 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.209023+0000", 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016292419999999999 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:71 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.54460129799998, 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098160388000000001, 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 71 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862253+0000", 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.862274+0000", 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.868833+0000", 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006558866 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869092+0000", 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025909700000000002 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.869345+0000", 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025318299999999998 2024-06-03T05:30:20.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.870885+0000", 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001540612 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.872594+0000", 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017081209999999999 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.954686+0000", 2024-06-03T05:30:20.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082092323999999994 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.958661+0000", 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039752340000000002 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:00.960414+0000", 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017523949999999999 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:82 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.29595097700002, 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13441932100000001, 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.506 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110904+0000", 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.110923+0000", 2024-06-03T05:30:20.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9001e-05 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111201+0000", 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027879400000000001 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111456+0000", 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025504299999999999 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.111708+0000", 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025164700000000003 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.121094+0000", 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093859540000000002 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.123457+0000", 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002363038 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.240810+0000", 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11735332699999999 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.242476+0000", 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001665899 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245323+0000", 2024-06-03T05:30:20.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002846618 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:83 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.29470558999998, 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.137790829, 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.511 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112149+0000", 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112163+0000", 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4438e-05 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112364+0000", 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020088699999999999 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112583+0000", 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021907400000000001 2024-06-03T05:30:20.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.112815+0000", 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023218499999999999 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.131317+0000", 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018501752999999999 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.133072+0000", 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017544609999999999 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.245566+0000", 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11249424099999999 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.247177+0000", 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001610769 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.249940+0000", 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027630210000000001 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.29364608100002, 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.151471354, 2024-06-03T05:30:20.516 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113208+0000", 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.113223+0000", 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.120108+0000", 2024-06-03T05:30:20.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068858449999999998 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145332+0000", 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025223619999999999 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.145595+0000", 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026325900000000001 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.150351+0000", 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047551449999999997 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.152098+0000", 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001747707 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.260071+0000", 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107973207 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.261838+0000", 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001767076 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264680+0000", 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028412580000000002 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.521 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.26079768599999, 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123180605, 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 85 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146057+0000", 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146075+0000", 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8159999999999999e-05 2024-06-03T05:30:20.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146316+0000", 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000241215 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146552+0000", 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023628000000000001 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.146784+0000", 2024-06-03T05:30:20.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023122299999999999 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.171194+0000", 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024410424 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.172939+0000", 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017447039999999999 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.264927+0000", 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.091988166999999996 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.266501+0000", 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015737119999999999 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269237+0000", 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027367200000000002 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.25971255500002, 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12660637199999999, 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 86 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147142+0000", 2024-06-03T05:30:20.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147156+0000", 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4136e-05 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147359+0000", 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020308500000000001 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147577+0000", 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000218135 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.147808+0000", 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023051699999999999 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.180608+0000", 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032800634000000002 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.182998+0000", 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389164 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.269462+0000", 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086464587999999995 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.271038+0000", 2024-06-03T05:30:20.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015755529999999999 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273748+0000", 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027105599999999999 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.25869553699999, 2024-06-03T05:30:20.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.130098677, 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 87 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148159+0000", 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.148173+0000", 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4319e-05 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.166932+0000", 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018758512000000001 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167226+0000", 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029416000000000002 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167496+0000", 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026982800000000001 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.192758+0000", 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025262508999999999 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.195598+0000", 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028393469999999999 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.273980+0000", 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078382747000000003 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.275545+0000", 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001564569 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278258+0000", 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027126860000000002 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-06-03T05:30:20.537 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 379.23891201999999, 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11470016500000001, 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167942+0000", 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.167962+0000", 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9704e-05 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168180+0000", 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021734900000000001 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168394+0000", 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021436299999999999 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.168636+0000", 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024209599999999999 2024-06-03T05:30:20.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.222791+0000", 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054155211000000002 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.224531+0000", 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017399780000000001 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.278483+0000", 2024-06-03T05:30:20.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053951462999999998 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.280022+0000", 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015393220000000001 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:01.282643+0000", 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026206789999999999 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:104 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 0~512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 375.114304027, 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.112050832, 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 104 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292550+0000", 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292564+0000", 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3399e-05 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292743+0000", 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000179531 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.292922+0000", 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017868699999999999 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293086+0000", 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016355499999999999 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.295217+0000", 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021310330000000001 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.297000+0000", 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001782905 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.401222+0000", 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104222892 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.402866+0000", 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016437299999999999 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404601+0000", 2024-06-03T05:30:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017351000000000001 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:105 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 375.113484363, 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.114002119, 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 105 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293370+0000", 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293381+0000", 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0991e-05 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293524+0000", 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142509 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293676+0000", 2024-06-03T05:30:20.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152755 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.293831+0000", 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015473700000000001 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.304406+0000", 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010574736 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.306188+0000", 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017818510000000001 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.404753+0000", 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098565669999999994 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.405721+0000", 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096761899999999997 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.407372+0000", 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016512510000000001 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4158.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 375.11278341399998, 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.119581228, 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4158", 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1302473185", 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 106 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294071+0000", 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.294081+0000", 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9869999999999994e-06 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323432+0000", 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029350497999999999 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323639+0000", 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020769000000000001 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.323812+0000", 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172817 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.326515+0000", 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027029279999999998 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.328306+0000", 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017914949999999999 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.410614+0000", 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082307593999999998 2024-06-03T05:30:20.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.411945+0000", 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00133055 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:05.413652+0000", 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001707669 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPull(1.0 e20/18 cost 1049576)", 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 368.70572968200003, 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11061585, 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701125+0000", 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.701134+0000", 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8100000000000004e-06 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.721705+0000", 2024-06-03T05:30:20.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020571011 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722330+0000", 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062519099999999998 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.722799+0000", 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046945199999999998 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.726612+0000", 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003812546 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:11.811741+0000", 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085128839999999997 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:20.562 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:20.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:30:20.564 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:20.564+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:20.664 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:20.665+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:20.738 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:20.738 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:20.738 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:20.738 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:20.747 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:30:20.765 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:20.766+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:20.823 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:24.633 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:20.867 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:20.867+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:20.880 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:24.691 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:20.881 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:20.881 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:20.881 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:20.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:30:20.967 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:20.967+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:21.029 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24)", 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 362.15608749500001, 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095595824999999995, 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.867983+0000", 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.868188+0000", 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000205381 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.874015+0000", 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058267830000000003 2024-06-03T05:30:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.878395+0000", 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043798800000000001 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.887756+0000", 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093607269999999992 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.907898+0000", 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020141991000000001 2024-06-03T05:30:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:24:18.963579+0000", 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055681063000000003 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4164.0:6 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 242.038128349, 2024-06-03T05:30:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035857200000000001, 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4164", 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.38:0/1314613108", 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 6 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985942+0000", 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.985987+0000", 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4521000000000003e-05 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986178+0000", 2024-06-03T05:30:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191428 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986373+0000", 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195059 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.986562+0000", 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018886700000000001 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.987875+0000", 2024-06-03T05:30:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013128599999999999 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-06-03T05:26:18.989528+0000", 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001652985 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-06-03T05:30:21.038 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:21.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:30:21.083 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:21.083+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:21.169 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:21.169+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:21.177 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:21.177 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:21.177 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:21.177 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:21.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:30:21.225 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:25.035 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:21.270 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:21.270+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:21.360 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:21.360 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:21.360 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:21.360 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:21.360 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:21.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:30:21.370 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:21.371+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:21.472 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:21.472+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:21.502 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:30:21.503 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:21.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:30:21.596 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:21.596+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:21.674 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:21.674+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:21.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:30:21.686 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:21.686 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:21.686 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:30:21.686 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:21.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:30:21.780 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:21.780+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:21.876 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:21.876+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:21.904 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:21.904 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:30:21.904 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-06-03T05:30:21.904 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-06-03T05:30:21.904 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:21.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:30:22.000 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:22.000+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:22.078 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:22.078+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:22.088 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:30:22.088 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-06-03T05:30:22.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-06-03T05:30:22.089 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-06-03T05:30:22.089 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:30:22.097 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-06-03T05:30:22.179 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:22.179+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:22.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-03T05:30:22.243 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:22.243 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:22.243 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-06-03T05:30:22.243 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:22.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-06-03T05:30:22.298 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:22.298+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:22.380 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:22.380+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:22.425 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-06-03T05:30:22.426 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-03T05:30:22.426 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 5, 7, 6, 4] 2024-06-03T05:30:22.426 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-06-03T05:30:22.426 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:30:22.438 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:22.428+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35175) UID: 0 2024-06-03T05:30:22.438 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:30:22.428+0000 b485640 -1 osd.5 102 *** Got signal Terminated *** 2024-06-03T05:30:22.454 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:30:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-06-03T05:30:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-06-03T05:30:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-06-03T05:30:22.454 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:30:22.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-06-03T05:30:22.482 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:22.481+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:22.531 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-06-03T05:30:22.531 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-03T05:30:22.582 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:22.582+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:22.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-06-03T05:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-06-03T05:30:22.625 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-03T05:30:22.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-06-03T05:30:22.683 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:22.683+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:22.739 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-06-03T05:30:22.739 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-03T05:30:22.756 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:22.756 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:22.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-06-03T05:30:22.784 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:22.784+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:22.839 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-06-03T05:30:22.839 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-03T05:30:22.885 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:30:22.885+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:30:22.899 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:22.899 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:22.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-06-03T05:30:22.960 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-06-03T05:30:22.960 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-03T05:30:22.986 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:30:22.986+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:30:23.044 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:23.044 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:23.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-06-03T05:30:23.055 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-06-03T05:30:23.055 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-03T05:30:23.086 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:30:23.087+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:30:23.156 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-06-03T05:30:23.156 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-03T05:30:23.188 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:30:23.188+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:30:23.190 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:23.190 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:23.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-06-03T05:30:23.250 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-06-03T05:30:23.250 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-03T05:30:23.309 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:30:23.310 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:30:23.310+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:30:23.373 INFO:teuthology.orchestra.run.smithi038.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","modified":"2024-06-03T05:22:16.844560Z","created":"2024-06-03T05:22:16.844560Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3301","nonce":0},{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-06-03T05:30:23.389 INFO:tasks.mon_recovery.ceph_manager:quorum is size 3 2024-06-03T05:30:23.390 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-06-03T05:30:23.400 INFO:tasks.thrashosds:joining thrashosds 2024-06-03T05:30:23.401 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:30:23.389+0000 b485640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:30:23.433 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:23.433 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:23.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-06-03T05:30:23.577 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:23.577 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:23.586 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-06-03T05:30:23.719 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:23.719 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:23.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-06-03T05:30:23.863 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:23.863 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:23.872 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-06-03T05:30:24.006 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:24.007 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:24.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-06-03T05:30:24.160 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:24.160 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:24.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-06-03T05:30:24.318 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:24.318 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:24.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-06-03T05:30:24.461 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:24.461 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:24.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-06-03T05:30:24.606 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:24.606 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:24.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-06-03T05:30:24.748 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:24.748 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:24.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-06-03T05:30:24.845 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:02:41.487 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:24.848 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:02:41.490 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:24.887 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:02:41.530 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:24.894 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:24.894 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:24.896 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:28.706 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:24.898 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:28.709 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:24.901 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:38.180 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:24.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-06-03T05:30:24.904 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:38.182 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:24.913 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:28.724 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:24.941 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:38.220 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:25.039 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:25.040 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:25.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-06-03T05:30:25.248 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:25.248 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:25.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-06-03T05:30:25.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-06-03T05:30:25.401 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-06-03T05:30:25.402 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-03T05:30:26.914 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-06-03T05:30:26.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-06-03T05:30:27.942 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-06-03T05:30:44.809 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:03:01.450 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:44.810 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:03:01.453 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:44.836 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:03:01.478 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:44.859 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:58.137 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:44.860 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:58.139 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:44.868 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:48.678 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:44.870 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:48.680 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:44.875 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:01:58.153 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:30:44.886 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:00:48.696 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:04.873 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:03:21.515 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:04.877 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:03:21.519 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:04.890 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:03:21.532 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:04.926 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:18.205 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:04.927 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:08.738 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:04.929 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:18.207 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:04.929 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:08.740 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:04.942 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:18.221 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:04.943 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:08.753 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:24.897 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:03:41.539 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:24.948 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:38.226 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:24.950 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:03:41.592 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:24.951 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:38.230 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:24.964 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:38.243 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:24.966 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:03:41.608 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:25.003 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:28.814 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:25.005 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:28.815 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:25.019 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:28.830 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:34.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-06-03T05:31:35.643 INFO:tasks.ceph.osd.5:Stopped 2024-06-03T05:31:35.643 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-06-03T05:31:35.644 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:31:35.644 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-06-03T05:31:35.663 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-06-03T05:31:35.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-06-03T05:31:37.321 INFO:tasks.thrashosds.thrasher:No PGs found for osd.5 2024-06-03T05:31:40.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-06-03T05:31:42.322 INFO:tasks.thrashosds.thrasher:reviving osd 2024-06-03T05:31:42.322 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-06-03T05:31:42.322 INFO:tasks.ceph.osd.5:Restarting daemon 2024-06-03T05:31:42.322 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-06-03T05:31:42.325 INFO:tasks.ceph.osd.5:Started 2024-06-03T05:31:42.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:31:42.435 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-06-03T05:31:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:31:42.438 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-06-03T05:31:45.005 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:04:01.647 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:45.006 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:04:01.649 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:45.020 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:04:01.662 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:45.058 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:48.868 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:45.059 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:58.337 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:45.059 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:48.870 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:45.061 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:58.339 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:45.074 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:01:48.884 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:45.075 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:02:58.354 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:31:47.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:31:47.535 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-06-03T05:31:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:31:47.538 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-06-03T05:31:49.566 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:07.167 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:49.632 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:07.234 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:50.203 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:31:50.197+0000 5dce500 -1 Falling back to public interface 2024-06-03T05:31:50.634 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:08.235 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:51.001 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:08.603 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:51.316 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:08.918 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:51.582 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:09.182 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:51.849 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:09.450 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:52.114 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:09.716 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:52.383 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:09.985 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:52.539 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:31:52.662 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:10.263 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:52.757 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:31:52.757 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:31:52.757 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:31:52.757 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:31:52.757 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:31:52.757 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:31:52.757 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-06-03T05:31:52.757 INFO:teuthology.orchestra.run.smithi064.stderr:config diff get 2024-06-03T05:31:52.757 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-06-03T05:31:52.758 INFO:teuthology.orchestra.run.smithi064.stderr:config help [] 2024-06-03T05:31:52.758 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-06-03T05:31:52.758 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:31:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:31:52.759 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-06-03T05:31:53.014 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:10.616 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:53.493 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:11.095 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:53.549 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:11.151 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:31:57.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:31:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:31:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:31:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:31:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:31:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:31:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:31:58.679 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:31:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:31:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation histogram block [] [] 2024-06-03T05:31:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator score block 2024-06-03T05:31:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-06-03T05:31:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:31:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:31:58.682 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-06-03T05:31:58.746 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:16.347 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:32:03.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:32:03.799 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:32:03.799 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:config diff get 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:config help [] 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-06-03T05:32:03.800 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:32:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:32:03.802 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-06-03T05:32:03.993 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:21.594 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:32:03.997 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:21.598 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:32:05.051 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:04:21.693 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:05.053 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:04:21.695 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:05.066 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:04:21.708 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:05.099 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:03:18.377 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:05.102 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:03:18.380 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:05.104 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:08.915 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:05.107 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:08.917 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:05.118 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:03:18.397 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:05.120 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:08.931 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:07.898 INFO:tasks.ceph.osd.5.smithi064.stderr:==00:00:00:25.500 64845== Warning: unimplemented fcntl command: 1038 2024-06-03T05:32:08.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:32:10.668 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:32:10.668 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:32:10.669 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:32:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:32:10.671 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-06-03T05:32:15.168 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:32:15.161+0000 5dce500 -1 osd.5 102 log_to_monitors true 2024-06-03T05:32:15.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:32:16.110 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-06-03T05:32:16.111 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-06-03T05:32:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-03T05:32:16.113 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-06-03T05:32:17.102 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:32:17.089+0000 10260640 -1 osd.5 102 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T05:32:21.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-06-03T05:32:21.270 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-06-03T05:32:21.270 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-06-03T05:32:21.270 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-06-03T05:32:21.270 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-06-03T05:32:21.278 INFO:tasks.thrashosds.thrasher:inning osd 2024-06-03T05:32:21.279 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-06-03T05:32:21.279 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-06-03T05:32:21.976 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-06-03T05:32:21.967+0000 ffa9640 -1 mon.a@0(leader).osd e108 definitely_dead 0 2024-06-03T05:32:22.906 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.5. 2024-06-03T05:32:22.925 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-06-03T05:32:22.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-06-03T05:32:23.229 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a: {} 2024-06-03T05:32:23.229 INFO:teuthology.orchestra.run.smithi038.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-03T05:32:23.364 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b: {} 2024-06-03T05:32:23.364 INFO:teuthology.orchestra.run.smithi038.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-03T05:32:23.565 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c: {} 2024-06-03T05:32:23.566 INFO:teuthology.orchestra.run.smithi038.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-03T05:32:23.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-06-03T05:32:25.166 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-06-03T05:32:25.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-06-03T05:32:26.282 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-06-03T05:32:26.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-06-03T05:32:27.434 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-06-03T05:32:27.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-06-03T05:32:28.657 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-06-03T05:32:28.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-06-03T05:32:29.686 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.2 to 1 (10000) 2024-06-03T05:32:29.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-06-03T05:32:30.294 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:04:46.936 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:30.299 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:04:46.941 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:30.320 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:04:46.962 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:30.349 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:03:43.628 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:30.351 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:34.161 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:30.354 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:34.165 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:30.358 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:03:43.636 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:30.378 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:03:43.657 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:30.382 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:34.193 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:31.322 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-06-03T05:32:31.337 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-06-03T05:32:32.885 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-06-03T05:32:32.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-06-03T05:32:33.896 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-06-03T05:32:33.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-06-03T05:32:35.104 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-06-03T05:32:35.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-06-03T05:32:36.207 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-06-03T05:32:36.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-06-03T05:32:37.365 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.6 to 1 (10000) 2024-06-03T05:32:37.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-06-03T05:32:38.507 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-06-03T05:32:38.521 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-06-03T05:32:39.650 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.4 to 1 (10000) 2024-06-03T05:32:39.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-06-03T05:32:40.805 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-06-03T05:32:40.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-06-03T05:32:41.824 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.5 to 1 (10000) 2024-06-03T05:32:41.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-06-03T05:32:42.903 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-06-03T05:32:42.919 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-06-03T05:32:42.919 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-06-03T05:32:42.919 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-06-03T05:32:42.919 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-06-03T05:32:42.919 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-06-03T05:32:42.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T05:32:43.666 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:32:43.666 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":127,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:32:42.768237+0000","last_up_change":"2024-06-03T05:32:17.633473+0000","last_in_change":"2024-06-03T05:32:22.763704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T05:23:56.753149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-03T05:24:12.898324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":905141127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":905141127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":905141127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":905141127}]},"public_addr":"172.21.15.38:6809/905141127","cluster_addr":"172.21.15.38:6811/905141127","heartbeat_back_addr":"172.21.15.38:6816/905141127","heartbeat_front_addr":"172.21.15.38:6813/905141127","state":["exists","up"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3490615740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3490615740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3490615740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3490615740}]},"public_addr":"172.21.15.38:6810/3490615740","cluster_addr":"172.21.15.38:6812/3490615740","heartbeat_back_addr":"172.21.15.38:6815/3490615740","heartbeat_front_addr":"172.21.15.38:6814/3490615740","state":["exists","up"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":3359326199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3359326199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3359326199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":3359326199}]},"public_addr":"172.21.15.38:6805/3359326199","cluster_addr":"172.21.15.38:6806/3359326199","heartbeat_back_addr":"172.21.15.38:6808/3359326199","heartbeat_front_addr":"172.21.15.38:6807/3359326199","state":["exists","up"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":2289724788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2289724788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2289724788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2289724788}]},"public_addr":"172.21.15.38:6801/2289724788","cluster_addr":"172.21.15.38:6802/2289724788","heartbeat_back_addr":"172.21.15.38:6804/2289724788","heartbeat_front_addr":"172.21.15.38:6803/2289724788","state":["exists","up"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":38,"up_from":83,"up_thru":83,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2508784804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":2508784804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":2508784804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2508784804}]},"public_addr":"172.21.15.64:6808/2508784804","cluster_addr":"172.21.15.64:6809/2508784804","heartbeat_back_addr":"172.21.15.64:6811/2508784804","heartbeat_front_addr":"172.21.15.64:6810/2508784804","state":["exists","up"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":102,"up_from":108,"up_thru":109,"down_at":103,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1978476220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":1978476220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":1978476220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1978476220}]},"public_addr":"172.21.15.64:6800/1978476220","cluster_addr":"172.21.15.64:6801/1978476220","heartbeat_back_addr":"172.21.15.64:6803/1978476220","heartbeat_front_addr":"172.21.15.64:6802/1978476220","state":["exists","up"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":35,"up_from":71,"up_thru":72,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":324655640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":324655640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6807","nonce":324655640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":324655640}]},"public_addr":"172.21.15.64:6804/324655640","cluster_addr":"172.21.15.64:6805/324655640","heartbeat_back_addr":"172.21.15.64:6807/324655640","heartbeat_front_addr":"172.21.15.64:6806/324655640","state":["exists","up"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":370645108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":370645108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":370645108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":370645108}]},"public_addr":"172.21.15.64:6812/370645108","cluster_addr":"172.21.15.64:6813/370645108","heartbeat_back_addr":"172.21.15.64:6815/370645108","heartbeat_front_addr":"172.21.15.64:6814/370645108","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:56.081253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.642638+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.131979+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.147738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.907705+0000","dead_epoch":38},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.533669+0000","dead_epoch":102},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.310102+0000","dead_epoch":36},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.959957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:32:43.679 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-06-03T05:32:43.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T05:32:44.455 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:32:44.455 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":127,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:32:42.768237+0000","last_up_change":"2024-06-03T05:32:17.633473+0000","last_in_change":"2024-06-03T05:32:22.763704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T05:23:56.753149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-03T05:24:12.898324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":905141127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":905141127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":905141127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":905141127}]},"public_addr":"172.21.15.38:6809/905141127","cluster_addr":"172.21.15.38:6811/905141127","heartbeat_back_addr":"172.21.15.38:6816/905141127","heartbeat_front_addr":"172.21.15.38:6813/905141127","state":["exists","up"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3490615740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3490615740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3490615740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3490615740}]},"public_addr":"172.21.15.38:6810/3490615740","cluster_addr":"172.21.15.38:6812/3490615740","heartbeat_back_addr":"172.21.15.38:6815/3490615740","heartbeat_front_addr":"172.21.15.38:6814/3490615740","state":["exists","up"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":3359326199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3359326199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3359326199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":3359326199}]},"public_addr":"172.21.15.38:6805/3359326199","cluster_addr":"172.21.15.38:6806/3359326199","heartbeat_back_addr":"172.21.15.38:6808/3359326199","heartbeat_front_addr":"172.21.15.38:6807/3359326199","state":["exists","up"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":2289724788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2289724788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2289724788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2289724788}]},"public_addr":"172.21.15.38:6801/2289724788","cluster_addr":"172.21.15.38:6802/2289724788","heartbeat_back_addr":"172.21.15.38:6804/2289724788","heartbeat_front_addr":"172.21.15.38:6803/2289724788","state":["exists","up"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":38,"up_from":83,"up_thru":83,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2508784804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":2508784804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":2508784804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2508784804}]},"public_addr":"172.21.15.64:6808/2508784804","cluster_addr":"172.21.15.64:6809/2508784804","heartbeat_back_addr":"172.21.15.64:6811/2508784804","heartbeat_front_addr":"172.21.15.64:6810/2508784804","state":["exists","up"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":102,"up_from":108,"up_thru":109,"down_at":103,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1978476220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":1978476220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":1978476220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1978476220}]},"public_addr":"172.21.15.64:6800/1978476220","cluster_addr":"172.21.15.64:6801/1978476220","heartbeat_back_addr":"172.21.15.64:6803/1978476220","heartbeat_front_addr":"172.21.15.64:6802/1978476220","state":["exists","up"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":35,"up_from":71,"up_thru":72,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":324655640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":324655640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6807","nonce":324655640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":324655640}]},"public_addr":"172.21.15.64:6804/324655640","cluster_addr":"172.21.15.64:6805/324655640","heartbeat_back_addr":"172.21.15.64:6807/324655640","heartbeat_front_addr":"172.21.15.64:6806/324655640","state":["exists","up"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":370645108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":370645108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":370645108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":370645108}]},"public_addr":"172.21.15.64:6812/370645108","cluster_addr":"172.21.15.64:6813/370645108","heartbeat_back_addr":"172.21.15.64:6815/370645108","heartbeat_front_addr":"172.21.15.64:6814/370645108","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:56.081253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.642638+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.131979+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.147738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.907705+0000","dead_epoch":38},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.533669+0000","dead_epoch":102},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.310102+0000","dead_epoch":36},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.959957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:32:44.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-06-03T05:32:44.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-06-03T05:32:44.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-06-03T05:32:44.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-06-03T05:32:44.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-06-03T05:32:44.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-06-03T05:32:44.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-06-03T05:32:44.472 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-06-03T05:32:44.827 INFO:teuthology.orchestra.run.smithi038.stdout:356482285606 2024-06-03T05:32:44.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-03T05:32:44.835 INFO:teuthology.orchestra.run.smithi038.stdout:77309411440 2024-06-03T05:32:44.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-03T05:32:44.881 INFO:teuthology.orchestra.run.smithi038.stdout:463856467976 2024-06-03T05:32:44.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-03T05:32:44.893 INFO:teuthology.orchestra.run.smithi038.stdout:77309411439 2024-06-03T05:32:44.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-03T05:32:44.907 INFO:teuthology.orchestra.run.smithi038.stdout:77309411438 2024-06-03T05:32:44.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-03T05:32:44.926 INFO:teuthology.orchestra.run.smithi038.stdout:304942678065 2024-06-03T05:32:44.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-06-03T05:32:44.940 INFO:teuthology.orchestra.run.smithi038.stdout:81604378736 2024-06-03T05:32:44.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-03T05:32:44.951 INFO:teuthology.orchestra.run.smithi038.stdout:68719476846 2024-06-03T05:32:44.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-03T05:32:45.676 INFO:teuthology.orchestra.run.smithi038.stdout:356482285604 2024-06-03T05:32:45.694 INFO:tasks.ceph.ceph_manager.ceph:need seq 356482285606 got 356482285604 for osd.4 2024-06-03T05:32:45.747 INFO:teuthology.orchestra.run.smithi038.stdout:77309411439 2024-06-03T05:32:45.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411440 got 77309411439 for osd.1 2024-06-03T05:32:46.044 INFO:teuthology.orchestra.run.smithi038.stdout:304942678064 2024-06-03T05:32:46.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 304942678065 got 304942678064 for osd.6 2024-06-03T05:32:46.063 INFO:teuthology.orchestra.run.smithi038.stdout:463856467975 2024-06-03T05:32:46.073 INFO:teuthology.orchestra.run.smithi038.stdout:77309411436 2024-06-03T05:32:46.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 463856467976 got 463856467975 for osd.5 2024-06-03T05:32:46.088 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411438 got 77309411436 for osd.7 2024-06-03T05:32:46.195 INFO:teuthology.orchestra.run.smithi038.stdout:68719476845 2024-06-03T05:32:46.210 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476846 got 68719476845 for osd.0 2024-06-03T05:32:46.277 INFO:teuthology.orchestra.run.smithi038.stdout:81604378735 2024-06-03T05:32:46.291 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378736 got 81604378735 for osd.3 2024-06-03T05:32:46.416 INFO:teuthology.orchestra.run.smithi038.stdout:77309411438 2024-06-03T05:32:46.429 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411439 got 77309411438 for osd.2 2024-06-03T05:32:46.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-03T05:32:46.763 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-03T05:32:47.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-06-03T05:32:47.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-03T05:32:47.088 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-03T05:32:47.211 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-03T05:32:47.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-03T05:32:47.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-03T05:32:47.547 INFO:teuthology.orchestra.run.smithi038.stdout:356482285605 2024-06-03T05:32:47.567 INFO:tasks.ceph.ceph_manager.ceph:need seq 356482285606 got 356482285605 for osd.4 2024-06-03T05:32:47.665 INFO:teuthology.orchestra.run.smithi038.stdout:77309411439 2024-06-03T05:32:47.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411440 got 77309411439 for osd.1 2024-06-03T05:32:47.925 INFO:teuthology.orchestra.run.smithi038.stdout:463856467976 2024-06-03T05:32:47.938 INFO:tasks.ceph.ceph_manager.ceph:need seq 463856467976 got 463856467976 for osd.5 2024-06-03T05:32:47.938 DEBUG:teuthology.parallel:result is None 2024-06-03T05:32:47.940 INFO:teuthology.orchestra.run.smithi038.stdout:304942678065 2024-06-03T05:32:47.954 INFO:tasks.ceph.ceph_manager.ceph:need seq 304942678065 got 304942678065 for osd.6 2024-06-03T05:32:47.954 DEBUG:teuthology.parallel:result is None 2024-06-03T05:32:48.142 INFO:teuthology.orchestra.run.smithi038.stdout:77309411438 2024-06-03T05:32:48.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411438 got 77309411438 for osd.7 2024-06-03T05:32:48.157 DEBUG:teuthology.parallel:result is None 2024-06-03T05:32:48.246 INFO:teuthology.orchestra.run.smithi038.stdout:81604378737 2024-06-03T05:32:48.259 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378736 got 81604378737 for osd.3 2024-06-03T05:32:48.259 DEBUG:teuthology.parallel:result is None 2024-06-03T05:32:48.277 INFO:teuthology.orchestra.run.smithi038.stdout:77309411440 2024-06-03T05:32:48.291 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411439 got 77309411440 for osd.2 2024-06-03T05:32:48.291 DEBUG:teuthology.parallel:result is None 2024-06-03T05:32:48.430 INFO:teuthology.orchestra.run.smithi038.stdout:68719476847 2024-06-03T05:32:48.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476846 got 68719476847 for osd.0 2024-06-03T05:32:48.442 DEBUG:teuthology.parallel:result is None 2024-06-03T05:32:48.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-03T05:32:48.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-03T05:32:49.244 INFO:teuthology.orchestra.run.smithi038.stdout:356482285606 2024-06-03T05:32:49.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 356482285606 got 356482285606 for osd.4 2024-06-03T05:32:49.263 DEBUG:teuthology.parallel:result is None 2024-06-03T05:32:49.358 INFO:teuthology.orchestra.run.smithi038.stdout:77309411441 2024-06-03T05:32:49.372 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411440 got 77309411441 for osd.1 2024-06-03T05:32:49.372 DEBUG:teuthology.parallel:result is None 2024-06-03T05:32:49.372 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-06-03T05:32:49.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:32:49.984 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:32:49.985 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:32:49.997 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-06-03T05:32:48.397600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239408,"kb_used_data":8512,"kb_used_omap":20,"kb_used_meta":218091,"kb_avail":754735312,"statfs":{"total":773094113280,"available":772848959488,"internally_reserved":0,"allocated":8716288,"data_stored":4516492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21175,"internal_metadata":223325513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.563094"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":220,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.187186+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:32:43.187186+0000","last_peered":"2024-06-03T05:32:43.187186+0000","last_clean":"2024-06-03T05:32:43.187186+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:32:43.187186+0000","last_undegraded":"2024-06-03T05:32:43.187186+0000","last_fullsized":"2024-06-03T05:32:43.187186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":106,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:46.328148+0000","last_change":"2024-06-03T05:29:45.112876+0000","last_active":"2024-06-03T05:32:46.328148+0000","last_peered":"2024-06-03T05:32:46.328148+0000","last_clean":"2024-06-03T05:32:46.328148+0000","last_became_active":"2024-06-03T05:29:45.083288+0000","last_became_peered":"2024-06-03T05:29:45.083288+0000","last_unstale":"2024-06-03T05:32:46.328148+0000","last_undegraded":"2024-06-03T05:32:46.328148+0000","last_fullsized":"2024-06-03T05:32:46.328148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:28:04.376061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":228,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.179892+0000","last_change":"2024-06-03T05:29:45.181097+0000","last_active":"2024-06-03T05:32:43.179892+0000","last_peered":"2024-06-03T05:32:43.179892+0000","last_clean":"2024-06-03T05:32:43.179892+0000","last_became_active":"2024-06-03T05:29:45.174231+0000","last_became_peered":"2024-06-03T05:29:45.174231+0000","last_unstale":"2024-06-03T05:32:43.179892+0000","last_undegraded":"2024-06-03T05:32:43.179892+0000","last_fullsized":"2024-06-03T05:32:43.179892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:02:58.418447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.623013+0000","last_change":"2024-06-03T05:32:25.005092+0000","last_active":"2024-06-03T05:32:47.623013+0000","last_peered":"2024-06-03T05:32:47.623013+0000","last_clean":"2024-06-03T05:32:47.623013+0000","last_became_active":"2024-06-03T05:32:24.975211+0000","last_became_peered":"2024-06-03T05:32:24.975211+0000","last_unstale":"2024-06-03T05:32:47.623013+0000","last_undegraded":"2024-06-03T05:32:47.623013+0000","last_fullsized":"2024-06-03T05:32:47.623013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:20:31.052690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":227,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.177875+0000","last_change":"2024-06-03T05:28:54.917780+0000","last_active":"2024-06-03T05:32:43.177875+0000","last_peered":"2024-06-03T05:32:43.177875+0000","last_clean":"2024-06-03T05:32:43.177875+0000","last_became_active":"2024-06-03T05:28:54.908410+0000","last_became_peered":"2024-06-03T05:28:54.908410+0000","last_unstale":"2024-06-03T05:32:43.177875+0000","last_undegraded":"2024-06-03T05:32:43.177875+0000","last_fullsized":"2024-06-03T05:32:43.177875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:52:54.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":231,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.795268+0000","last_change":"2024-06-03T05:32:24.559256+0000","last_active":"2024-06-03T05:32:43.795268+0000","last_peered":"2024-06-03T05:32:43.795268+0000","last_clean":"2024-06-03T05:32:43.795268+0000","last_became_active":"2024-06-03T05:32:24.553004+0000","last_became_peered":"2024-06-03T05:32:24.553004+0000","last_unstale":"2024-06-03T05:32:43.795268+0000","last_undegraded":"2024-06-03T05:32:43.795268+0000","last_fullsized":"2024-06-03T05:32:43.795268+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:20:48.090948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":234,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.174546+0000","last_change":"2024-06-03T05:32:24.556287+0000","last_active":"2024-06-03T05:32:43.174546+0000","last_peered":"2024-06-03T05:32:43.174546+0000","last_clean":"2024-06-03T05:32:43.174546+0000","last_became_active":"2024-06-03T05:32:24.549787+0000","last_became_peered":"2024-06-03T05:32:24.549787+0000","last_unstale":"2024-06-03T05:32:43.174546+0000","last_undegraded":"2024-06-03T05:32:43.174546+0000","last_fullsized":"2024-06-03T05:32:43.174546+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:05:24.230660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":221,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.183588+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:32:43.183588+0000","last_peered":"2024-06-03T05:32:43.183588+0000","last_clean":"2024-06-03T05:32:43.183588+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:32:43.183588+0000","last_undegraded":"2024-06-03T05:32:43.183588+0000","last_fullsized":"2024-06-03T05:32:43.183588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":250,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:42.930928+0000","last_change":"2024-06-03T05:32:24.938360+0000","last_active":"2024-06-03T05:32:42.930928+0000","last_peered":"2024-06-03T05:32:42.930928+0000","last_clean":"2024-06-03T05:32:42.930928+0000","last_became_active":"2024-06-03T05:32:24.556417+0000","last_became_peered":"2024-06-03T05:32:24.556417+0000","last_unstale":"2024-06-03T05:32:42.930928+0000","last_undegraded":"2024-06-03T05:32:42.930928+0000","last_fullsized":"2024-06-03T05:32:42.930928+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:55:33.938265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204224,"data_stored":1193520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378737,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":987136,"data_stored":464021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411438,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":464021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":83,"seq":356482285607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27707,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28372686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":464021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28380,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343460,"statfs":{"total":96636764160,"available":96607703040,"internally_reserved":0,"allocated":1388544,"data_stored":861861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476847,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":987136,"data_stored":464021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":71,"seq":304942678066,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28503763},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":108,"seq":463856467977,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1356,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1388544,"data_stored":861861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28176072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:32:49.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:32:50.509 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:05:07.151 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:50.512 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:05:07.155 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:50.535 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:05:07.177 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:50.559 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:54.369 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:50.562 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:04:03.841 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:50.565 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:04:03.843 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:50.567 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:54.377 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:50.586 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:04:03.865 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:50.600 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:02:54.411 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:32:50.637 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:32:50.637 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:32:50.651 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-06-03T05:32:50.397902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239432,"kb_used_data":8544,"kb_used_omap":20,"kb_used_meta":218091,"kb_avail":754735288,"statfs":{"total":773094113280,"available":772848934912,"internally_reserved":0,"allocated":8749056,"data_stored":4533780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21175,"internal_metadata":223325513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001244"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":220,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.187186+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:32:43.187186+0000","last_peered":"2024-06-03T05:32:43.187186+0000","last_clean":"2024-06-03T05:32:43.187186+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:32:43.187186+0000","last_undegraded":"2024-06-03T05:32:43.187186+0000","last_fullsized":"2024-06-03T05:32:43.187186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":106,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:46.328148+0000","last_change":"2024-06-03T05:29:45.112876+0000","last_active":"2024-06-03T05:32:46.328148+0000","last_peered":"2024-06-03T05:32:46.328148+0000","last_clean":"2024-06-03T05:32:46.328148+0000","last_became_active":"2024-06-03T05:29:45.083288+0000","last_became_peered":"2024-06-03T05:29:45.083288+0000","last_unstale":"2024-06-03T05:32:46.328148+0000","last_undegraded":"2024-06-03T05:32:46.328148+0000","last_fullsized":"2024-06-03T05:32:46.328148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:28:04.376061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":228,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.179892+0000","last_change":"2024-06-03T05:29:45.181097+0000","last_active":"2024-06-03T05:32:43.179892+0000","last_peered":"2024-06-03T05:32:43.179892+0000","last_clean":"2024-06-03T05:32:43.179892+0000","last_became_active":"2024-06-03T05:29:45.174231+0000","last_became_peered":"2024-06-03T05:29:45.174231+0000","last_unstale":"2024-06-03T05:32:43.179892+0000","last_undegraded":"2024-06-03T05:32:43.179892+0000","last_fullsized":"2024-06-03T05:32:43.179892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:02:58.418447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.623013+0000","last_change":"2024-06-03T05:32:25.005092+0000","last_active":"2024-06-03T05:32:47.623013+0000","last_peered":"2024-06-03T05:32:47.623013+0000","last_clean":"2024-06-03T05:32:47.623013+0000","last_became_active":"2024-06-03T05:32:24.975211+0000","last_became_peered":"2024-06-03T05:32:24.975211+0000","last_unstale":"2024-06-03T05:32:47.623013+0000","last_undegraded":"2024-06-03T05:32:47.623013+0000","last_fullsized":"2024-06-03T05:32:47.623013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:20:31.052690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":227,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.177875+0000","last_change":"2024-06-03T05:28:54.917780+0000","last_active":"2024-06-03T05:32:43.177875+0000","last_peered":"2024-06-03T05:32:43.177875+0000","last_clean":"2024-06-03T05:32:43.177875+0000","last_became_active":"2024-06-03T05:28:54.908410+0000","last_became_peered":"2024-06-03T05:28:54.908410+0000","last_unstale":"2024-06-03T05:32:43.177875+0000","last_undegraded":"2024-06-03T05:32:43.177875+0000","last_fullsized":"2024-06-03T05:32:43.177875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:52:54.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":233,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.614380+0000","last_change":"2024-06-03T05:32:24.559256+0000","last_active":"2024-06-03T05:32:47.614380+0000","last_peered":"2024-06-03T05:32:47.614380+0000","last_clean":"2024-06-03T05:32:47.614380+0000","last_became_active":"2024-06-03T05:32:24.553004+0000","last_became_peered":"2024-06-03T05:32:24.553004+0000","last_unstale":"2024-06-03T05:32:47.614380+0000","last_undegraded":"2024-06-03T05:32:47.614380+0000","last_fullsized":"2024-06-03T05:32:47.614380+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:20:48.090948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":234,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.174546+0000","last_change":"2024-06-03T05:32:24.556287+0000","last_active":"2024-06-03T05:32:43.174546+0000","last_peered":"2024-06-03T05:32:43.174546+0000","last_clean":"2024-06-03T05:32:43.174546+0000","last_became_active":"2024-06-03T05:32:24.549787+0000","last_became_peered":"2024-06-03T05:32:24.549787+0000","last_unstale":"2024-06-03T05:32:43.174546+0000","last_undegraded":"2024-06-03T05:32:43.174546+0000","last_fullsized":"2024-06-03T05:32:43.174546+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:05:24.230660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":221,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.183588+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:32:43.183588+0000","last_peered":"2024-06-03T05:32:43.183588+0000","last_clean":"2024-06-03T05:32:43.183588+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:32:43.183588+0000","last_undegraded":"2024-06-03T05:32:43.183588+0000","last_fullsized":"2024-06-03T05:32:43.183588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":252,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.610061+0000","last_change":"2024-06-03T05:32:24.938360+0000","last_active":"2024-06-03T05:32:47.610061+0000","last_peered":"2024-06-03T05:32:47.610061+0000","last_clean":"2024-06-03T05:32:47.610061+0000","last_became_active":"2024-06-03T05:32:24.556417+0000","last_became_peered":"2024-06-03T05:32:24.556417+0000","last_unstale":"2024-06-03T05:32:47.610061+0000","last_undegraded":"2024-06-03T05:32:47.610061+0000","last_fullsized":"2024-06-03T05:32:47.610061+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:55:33.938265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204224,"data_stored":1193520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378737,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":987136,"data_stored":464021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411439,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":83,"seq":356482285607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27707,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28372686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411442,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28384,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343456,"statfs":{"total":96636764160,"available":96607698944,"internally_reserved":0,"allocated":1396736,"data_stored":866183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476848,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":71,"seq":304942678066,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28503763},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":108,"seq":463856467977,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1356,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1388544,"data_stored":861861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28176072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:32:50.652 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-06-03T05:32:50.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-06-03T05:32:51.531 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-06-03T05:32:51.541 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-03T05:32:51.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:32:52.195 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:32:52.196 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:32:52.208 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-06-03T05:32:50.397902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239432,"kb_used_data":8544,"kb_used_omap":20,"kb_used_meta":218091,"kb_avail":754735288,"statfs":{"total":773094113280,"available":772848934912,"internally_reserved":0,"allocated":8749056,"data_stored":4533780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21175,"internal_metadata":223325513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001244"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":220,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.187186+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:32:43.187186+0000","last_peered":"2024-06-03T05:32:43.187186+0000","last_clean":"2024-06-03T05:32:43.187186+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:32:43.187186+0000","last_undegraded":"2024-06-03T05:32:43.187186+0000","last_fullsized":"2024-06-03T05:32:43.187186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":106,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:46.328148+0000","last_change":"2024-06-03T05:29:45.112876+0000","last_active":"2024-06-03T05:32:46.328148+0000","last_peered":"2024-06-03T05:32:46.328148+0000","last_clean":"2024-06-03T05:32:46.328148+0000","last_became_active":"2024-06-03T05:29:45.083288+0000","last_became_peered":"2024-06-03T05:29:45.083288+0000","last_unstale":"2024-06-03T05:32:46.328148+0000","last_undegraded":"2024-06-03T05:32:46.328148+0000","last_fullsized":"2024-06-03T05:32:46.328148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:28:04.376061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":228,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.179892+0000","last_change":"2024-06-03T05:29:45.181097+0000","last_active":"2024-06-03T05:32:43.179892+0000","last_peered":"2024-06-03T05:32:43.179892+0000","last_clean":"2024-06-03T05:32:43.179892+0000","last_became_active":"2024-06-03T05:29:45.174231+0000","last_became_peered":"2024-06-03T05:29:45.174231+0000","last_unstale":"2024-06-03T05:32:43.179892+0000","last_undegraded":"2024-06-03T05:32:43.179892+0000","last_fullsized":"2024-06-03T05:32:43.179892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:02:58.418447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.623013+0000","last_change":"2024-06-03T05:32:25.005092+0000","last_active":"2024-06-03T05:32:47.623013+0000","last_peered":"2024-06-03T05:32:47.623013+0000","last_clean":"2024-06-03T05:32:47.623013+0000","last_became_active":"2024-06-03T05:32:24.975211+0000","last_became_peered":"2024-06-03T05:32:24.975211+0000","last_unstale":"2024-06-03T05:32:47.623013+0000","last_undegraded":"2024-06-03T05:32:47.623013+0000","last_fullsized":"2024-06-03T05:32:47.623013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:20:31.052690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":227,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.177875+0000","last_change":"2024-06-03T05:28:54.917780+0000","last_active":"2024-06-03T05:32:43.177875+0000","last_peered":"2024-06-03T05:32:43.177875+0000","last_clean":"2024-06-03T05:32:43.177875+0000","last_became_active":"2024-06-03T05:28:54.908410+0000","last_became_peered":"2024-06-03T05:28:54.908410+0000","last_unstale":"2024-06-03T05:32:43.177875+0000","last_undegraded":"2024-06-03T05:32:43.177875+0000","last_fullsized":"2024-06-03T05:32:43.177875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:52:54.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":233,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.614380+0000","last_change":"2024-06-03T05:32:24.559256+0000","last_active":"2024-06-03T05:32:47.614380+0000","last_peered":"2024-06-03T05:32:47.614380+0000","last_clean":"2024-06-03T05:32:47.614380+0000","last_became_active":"2024-06-03T05:32:24.553004+0000","last_became_peered":"2024-06-03T05:32:24.553004+0000","last_unstale":"2024-06-03T05:32:47.614380+0000","last_undegraded":"2024-06-03T05:32:47.614380+0000","last_fullsized":"2024-06-03T05:32:47.614380+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:20:48.090948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":234,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.174546+0000","last_change":"2024-06-03T05:32:24.556287+0000","last_active":"2024-06-03T05:32:43.174546+0000","last_peered":"2024-06-03T05:32:43.174546+0000","last_clean":"2024-06-03T05:32:43.174546+0000","last_became_active":"2024-06-03T05:32:24.549787+0000","last_became_peered":"2024-06-03T05:32:24.549787+0000","last_unstale":"2024-06-03T05:32:43.174546+0000","last_undegraded":"2024-06-03T05:32:43.174546+0000","last_fullsized":"2024-06-03T05:32:43.174546+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:05:24.230660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":221,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-06-03T05:32:43.183588+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:32:43.183588+0000","last_peered":"2024-06-03T05:32:43.183588+0000","last_clean":"2024-06-03T05:32:43.183588+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:32:43.183588+0000","last_undegraded":"2024-06-03T05:32:43.183588+0000","last_fullsized":"2024-06-03T05:32:43.183588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":252,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.610061+0000","last_change":"2024-06-03T05:32:24.938360+0000","last_active":"2024-06-03T05:32:47.610061+0000","last_peered":"2024-06-03T05:32:47.610061+0000","last_clean":"2024-06-03T05:32:47.610061+0000","last_became_active":"2024-06-03T05:32:24.556417+0000","last_became_peered":"2024-06-03T05:32:24.556417+0000","last_unstale":"2024-06-03T05:32:47.610061+0000","last_undegraded":"2024-06-03T05:32:47.610061+0000","last_fullsized":"2024-06-03T05:32:47.610061+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:55:33.938265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204224,"data_stored":1193520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378737,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":987136,"data_stored":464021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411439,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":83,"seq":356482285607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27707,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28372686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411442,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28384,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343456,"statfs":{"total":96636764160,"available":96607698944,"internally_reserved":0,"allocated":1396736,"data_stored":866183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476848,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":71,"seq":304942678066,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28503763},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":108,"seq":463856467977,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1356,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1388544,"data_stored":861861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28176072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:32:52.209 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:32:52.809 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:32:52.809 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:32:52.822 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-06-03T05:32:52.398214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239440,"kb_used_data":8552,"kb_used_omap":20,"kb_used_meta":218091,"kb_avail":754735280,"statfs":{"total":773094113280,"available":772848926720,"internally_reserved":0,"allocated":8757248,"data_stored":4538102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21175,"internal_metadata":223325513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.483967"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":222,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.617418+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:32:47.617418+0000","last_peered":"2024-06-03T05:32:47.617418+0000","last_clean":"2024-06-03T05:32:47.617418+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:32:47.617418+0000","last_undegraded":"2024-06-03T05:32:47.617418+0000","last_fullsized":"2024-06-03T05:32:47.617418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":106,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:46.328148+0000","last_change":"2024-06-03T05:29:45.112876+0000","last_active":"2024-06-03T05:32:46.328148+0000","last_peered":"2024-06-03T05:32:46.328148+0000","last_clean":"2024-06-03T05:32:46.328148+0000","last_became_active":"2024-06-03T05:29:45.083288+0000","last_became_peered":"2024-06-03T05:29:45.083288+0000","last_unstale":"2024-06-03T05:32:46.328148+0000","last_undegraded":"2024-06-03T05:32:46.328148+0000","last_fullsized":"2024-06-03T05:32:46.328148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:28:04.376061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":230,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.630419+0000","last_change":"2024-06-03T05:29:45.181097+0000","last_active":"2024-06-03T05:32:47.630419+0000","last_peered":"2024-06-03T05:32:47.630419+0000","last_clean":"2024-06-03T05:32:47.630419+0000","last_became_active":"2024-06-03T05:29:45.174231+0000","last_became_peered":"2024-06-03T05:29:45.174231+0000","last_unstale":"2024-06-03T05:32:47.630419+0000","last_undegraded":"2024-06-03T05:32:47.630419+0000","last_fullsized":"2024-06-03T05:32:47.630419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:02:58.418447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.623013+0000","last_change":"2024-06-03T05:32:25.005092+0000","last_active":"2024-06-03T05:32:47.623013+0000","last_peered":"2024-06-03T05:32:47.623013+0000","last_clean":"2024-06-03T05:32:47.623013+0000","last_became_active":"2024-06-03T05:32:24.975211+0000","last_became_peered":"2024-06-03T05:32:24.975211+0000","last_unstale":"2024-06-03T05:32:47.623013+0000","last_undegraded":"2024-06-03T05:32:47.623013+0000","last_fullsized":"2024-06-03T05:32:47.623013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:20:31.052690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":229,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.623872+0000","last_change":"2024-06-03T05:28:54.917780+0000","last_active":"2024-06-03T05:32:47.623872+0000","last_peered":"2024-06-03T05:32:47.623872+0000","last_clean":"2024-06-03T05:32:47.623872+0000","last_became_active":"2024-06-03T05:28:54.908410+0000","last_became_peered":"2024-06-03T05:28:54.908410+0000","last_unstale":"2024-06-03T05:32:47.623872+0000","last_undegraded":"2024-06-03T05:32:47.623872+0000","last_fullsized":"2024-06-03T05:32:47.623872+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:52:54.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":233,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.614380+0000","last_change":"2024-06-03T05:32:24.559256+0000","last_active":"2024-06-03T05:32:47.614380+0000","last_peered":"2024-06-03T05:32:47.614380+0000","last_clean":"2024-06-03T05:32:47.614380+0000","last_became_active":"2024-06-03T05:32:24.553004+0000","last_became_peered":"2024-06-03T05:32:24.553004+0000","last_unstale":"2024-06-03T05:32:47.614380+0000","last_undegraded":"2024-06-03T05:32:47.614380+0000","last_fullsized":"2024-06-03T05:32:47.614380+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:20:48.090948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":236,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.620658+0000","last_change":"2024-06-03T05:32:24.556287+0000","last_active":"2024-06-03T05:32:47.620658+0000","last_peered":"2024-06-03T05:32:47.620658+0000","last_clean":"2024-06-03T05:32:47.620658+0000","last_became_active":"2024-06-03T05:32:24.549787+0000","last_became_peered":"2024-06-03T05:32:24.549787+0000","last_unstale":"2024-06-03T05:32:47.620658+0000","last_undegraded":"2024-06-03T05:32:47.620658+0000","last_fullsized":"2024-06-03T05:32:47.620658+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:05:24.230660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":223,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.627139+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:32:47.627139+0000","last_peered":"2024-06-03T05:32:47.627139+0000","last_clean":"2024-06-03T05:32:47.627139+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:32:47.627139+0000","last_undegraded":"2024-06-03T05:32:47.627139+0000","last_fullsized":"2024-06-03T05:32:47.627139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":252,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.610061+0000","last_change":"2024-06-03T05:32:24.938360+0000","last_active":"2024-06-03T05:32:47.610061+0000","last_peered":"2024-06-03T05:32:47.610061+0000","last_clean":"2024-06-03T05:32:47.610061+0000","last_became_active":"2024-06-03T05:32:24.556417+0000","last_became_peered":"2024-06-03T05:32:24.556417+0000","last_unstale":"2024-06-03T05:32:47.610061+0000","last_undegraded":"2024-06-03T05:32:47.610061+0000","last_fullsized":"2024-06-03T05:32:47.610061+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:55:33.938265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204224,"data_stored":1193520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378738,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411439,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":83,"seq":356482285607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27707,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28372686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411442,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28384,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343456,"statfs":{"total":96636764160,"available":96607698944,"internally_reserved":0,"allocated":1396736,"data_stored":866183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476848,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":71,"seq":304942678066,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28503763},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":108,"seq":463856467977,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1356,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1388544,"data_stored":861861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28176072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:32:52.822 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-03T05:32:52.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:32:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:32:53.424 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:32:53.436 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-06-03T05:32:52.398214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239440,"kb_used_data":8552,"kb_used_omap":20,"kb_used_meta":218091,"kb_avail":754735280,"statfs":{"total":773094113280,"available":772848926720,"internally_reserved":0,"allocated":8757248,"data_stored":4538102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21175,"internal_metadata":223325513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.483967"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":222,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.617418+0000","last_change":"2024-06-03T05:24:15.497424+0000","last_active":"2024-06-03T05:32:47.617418+0000","last_peered":"2024-06-03T05:32:47.617418+0000","last_clean":"2024-06-03T05:32:47.617418+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:32:47.617418+0000","last_undegraded":"2024-06-03T05:32:47.617418+0000","last_fullsized":"2024-06-03T05:32:47.617418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:04:21.513283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":106,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:46.328148+0000","last_change":"2024-06-03T05:29:45.112876+0000","last_active":"2024-06-03T05:32:46.328148+0000","last_peered":"2024-06-03T05:32:46.328148+0000","last_clean":"2024-06-03T05:32:46.328148+0000","last_became_active":"2024-06-03T05:29:45.083288+0000","last_became_peered":"2024-06-03T05:29:45.083288+0000","last_unstale":"2024-06-03T05:32:46.328148+0000","last_undegraded":"2024-06-03T05:32:46.328148+0000","last_fullsized":"2024-06-03T05:32:46.328148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:28:04.376061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":230,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.630419+0000","last_change":"2024-06-03T05:29:45.181097+0000","last_active":"2024-06-03T05:32:47.630419+0000","last_peered":"2024-06-03T05:32:47.630419+0000","last_clean":"2024-06-03T05:32:47.630419+0000","last_became_active":"2024-06-03T05:29:45.174231+0000","last_became_peered":"2024-06-03T05:29:45.174231+0000","last_unstale":"2024-06-03T05:32:47.630419+0000","last_undegraded":"2024-06-03T05:32:47.630419+0000","last_fullsized":"2024-06-03T05:32:47.630419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:02:58.418447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.623013+0000","last_change":"2024-06-03T05:32:25.005092+0000","last_active":"2024-06-03T05:32:47.623013+0000","last_peered":"2024-06-03T05:32:47.623013+0000","last_clean":"2024-06-03T05:32:47.623013+0000","last_became_active":"2024-06-03T05:32:24.975211+0000","last_became_peered":"2024-06-03T05:32:24.975211+0000","last_unstale":"2024-06-03T05:32:47.623013+0000","last_undegraded":"2024-06-03T05:32:47.623013+0000","last_fullsized":"2024-06-03T05:32:47.623013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:20:31.052690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":229,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.623872+0000","last_change":"2024-06-03T05:28:54.917780+0000","last_active":"2024-06-03T05:32:47.623872+0000","last_peered":"2024-06-03T05:32:47.623872+0000","last_clean":"2024-06-03T05:32:47.623872+0000","last_became_active":"2024-06-03T05:28:54.908410+0000","last_became_peered":"2024-06-03T05:28:54.908410+0000","last_unstale":"2024-06-03T05:32:47.623872+0000","last_undegraded":"2024-06-03T05:32:47.623872+0000","last_fullsized":"2024-06-03T05:32:47.623872+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:52:54.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":233,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.614380+0000","last_change":"2024-06-03T05:32:24.559256+0000","last_active":"2024-06-03T05:32:47.614380+0000","last_peered":"2024-06-03T05:32:47.614380+0000","last_clean":"2024-06-03T05:32:47.614380+0000","last_became_active":"2024-06-03T05:32:24.553004+0000","last_became_peered":"2024-06-03T05:32:24.553004+0000","last_unstale":"2024-06-03T05:32:47.614380+0000","last_undegraded":"2024-06-03T05:32:47.614380+0000","last_fullsized":"2024-06-03T05:32:47.614380+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:20:48.090948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":236,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.620658+0000","last_change":"2024-06-03T05:32:24.556287+0000","last_active":"2024-06-03T05:32:47.620658+0000","last_peered":"2024-06-03T05:32:47.620658+0000","last_clean":"2024-06-03T05:32:47.620658+0000","last_became_active":"2024-06-03T05:32:24.549787+0000","last_became_peered":"2024-06-03T05:32:24.549787+0000","last_unstale":"2024-06-03T05:32:47.620658+0000","last_undegraded":"2024-06-03T05:32:47.620658+0000","last_fullsized":"2024-06-03T05:32:47.620658+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:05:24.230660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":223,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.627139+0000","last_change":"2024-06-03T05:24:15.017637+0000","last_active":"2024-06-03T05:32:47.627139+0000","last_peered":"2024-06-03T05:32:47.627139+0000","last_clean":"2024-06-03T05:32:47.627139+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:32:47.627139+0000","last_undegraded":"2024-06-03T05:32:47.627139+0000","last_fullsized":"2024-06-03T05:32:47.627139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:58:10.132508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'76","reported_seq":252,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.610061+0000","last_change":"2024-06-03T05:32:24.938360+0000","last_active":"2024-06-03T05:32:47.610061+0000","last_peered":"2024-06-03T05:32:47.610061+0000","last_clean":"2024-06-03T05:32:47.610061+0000","last_became_active":"2024-06-03T05:32:24.556417+0000","last_became_peered":"2024-06-03T05:32:24.556417+0000","last_unstale":"2024-06-03T05:32:47.610061+0000","last_undegraded":"2024-06-03T05:32:47.610061+0000","last_fullsized":"2024-06-03T05:32:47.610061+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:23:57.189945+0000","last_clean_scrub_stamp":"2024-06-03T05:23:57.189945+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:55:33.938265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204224,"data_stored":1193520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378738,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411439,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":83,"seq":356482285607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27707,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28372686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411442,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28384,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343456,"statfs":{"total":96636764160,"available":96607698944,"internally_reserved":0,"allocated":1396736,"data_stored":866183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476848,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":71,"seq":304942678066,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28503763},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":108,"seq":463856467977,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1356,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1388544,"data_stored":861861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28176072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:32:53.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T05:32:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:32:54.082 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":128,"fsid":"ebc09f7e-f094-4421-826f-479fc526fc62","created":"2024-06-03T05:23:10.198378+0000","modified":"2024-06-03T05:32:45.137386+0000","last_up_change":"2024-06-03T05:32:17.633473+0000","last_in_change":"2024-06-03T05:32:22.763704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T05:23:56.753149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-03T05:24:12.898324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32b256d9-eee7-435b-8aec-fef2ba90366c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":905141127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":905141127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":905141127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":905141127}]},"public_addr":"172.21.15.38:6809/905141127","cluster_addr":"172.21.15.38:6811/905141127","heartbeat_back_addr":"172.21.15.38:6816/905141127","heartbeat_front_addr":"172.21.15.38:6813/905141127","state":["exists","up"]},{"osd":1,"uuid":"371c2e00-54db-430b-8497-a4c9ff6753b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3490615740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3490615740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3490615740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3490615740}]},"public_addr":"172.21.15.38:6810/3490615740","cluster_addr":"172.21.15.38:6812/3490615740","heartbeat_back_addr":"172.21.15.38:6815/3490615740","heartbeat_front_addr":"172.21.15.38:6814/3490615740","state":["exists","up"]},{"osd":2,"uuid":"e8145335-ec5d-4dd1-9250-a4ab5b159d29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":3359326199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":3359326199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3359326199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":3359326199}]},"public_addr":"172.21.15.38:6805/3359326199","cluster_addr":"172.21.15.38:6806/3359326199","heartbeat_back_addr":"172.21.15.38:6808/3359326199","heartbeat_front_addr":"172.21.15.38:6807/3359326199","state":["exists","up"]},{"osd":3,"uuid":"1f2ffdb2-03b9-4a21-b31e-0ffd682b5a6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":2289724788}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2289724788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2289724788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2289724788}]},"public_addr":"172.21.15.38:6801/2289724788","cluster_addr":"172.21.15.38:6802/2289724788","heartbeat_back_addr":"172.21.15.38:6804/2289724788","heartbeat_front_addr":"172.21.15.38:6803/2289724788","state":["exists","up"]},{"osd":4,"uuid":"46b46f54-04d0-46c0-809c-64fa385e6780","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":38,"up_from":83,"up_thru":83,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2508784804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6809","nonce":2508784804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":2508784804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2508784804}]},"public_addr":"172.21.15.64:6808/2508784804","cluster_addr":"172.21.15.64:6809/2508784804","heartbeat_back_addr":"172.21.15.64:6811/2508784804","heartbeat_front_addr":"172.21.15.64:6810/2508784804","state":["exists","up"]},{"osd":5,"uuid":"f7a50ac0-eba9-4602-b90a-2dbc4bb20593","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":102,"up_from":108,"up_thru":109,"down_at":103,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1978476220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":1978476220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":1978476220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1978476220}]},"public_addr":"172.21.15.64:6800/1978476220","cluster_addr":"172.21.15.64:6801/1978476220","heartbeat_back_addr":"172.21.15.64:6803/1978476220","heartbeat_front_addr":"172.21.15.64:6802/1978476220","state":["exists","up"]},{"osd":6,"uuid":"081024c0-2482-4846-a7bd-594a58f2ceef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":35,"up_from":71,"up_thru":72,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":324655640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":324655640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6807","nonce":324655640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":324655640}]},"public_addr":"172.21.15.64:6804/324655640","cluster_addr":"172.21.15.64:6805/324655640","heartbeat_back_addr":"172.21.15.64:6807/324655640","heartbeat_front_addr":"172.21.15.64:6806/324655640","state":["exists","up"]},{"osd":7,"uuid":"6234c757-8f7b-497e-9347-4a7983df592c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":370645108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6813","nonce":370645108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6815","nonce":370645108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":370645108}]},"public_addr":"172.21.15.64:6812/370645108","cluster_addr":"172.21.15.64:6813/370645108","heartbeat_back_addr":"172.21.15.64:6815/370645108","heartbeat_front_addr":"172.21.15.64:6814/370645108","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:56.081253+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.642638+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.131979+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:55.147738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.907705+0000","dead_epoch":38},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.533669+0000","dead_epoch":102},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.310102+0000","dead_epoch":36},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T05:23:54.959957+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T05:32:55.096 INFO:tasks.ceph:Scrubbing osd.0 2024-06-03T05:32:55.097 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-06-03T05:32:55.302 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:32:55.303 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T05:32:55.303 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:32:55.314 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-06-03T05:32:55.918 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 0 to deep-scrub 2024-06-03T05:32:55.931 INFO:tasks.ceph:Scrubbing osd.1 2024-06-03T05:32:55.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-06-03T05:32:56.133 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:32:56.133 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T05:32:56.133 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:32:56.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-06-03T05:32:56.761 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 1 to deep-scrub 2024-06-03T05:32:56.774 INFO:tasks.ceph:Scrubbing osd.2 2024-06-03T05:32:56.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-06-03T05:32:56.981 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:32:56.981 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T05:32:56.981 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:32:56.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-06-03T05:32:57.620 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 2 to deep-scrub 2024-06-03T05:32:57.634 INFO:tasks.ceph:Scrubbing osd.3 2024-06-03T05:32:57.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-06-03T05:32:57.849 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:32:57.849 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T05:32:57.849 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:32:57.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-06-03T05:32:58.474 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 3 to deep-scrub 2024-06-03T05:32:58.487 INFO:tasks.ceph:Scrubbing osd.4 2024-06-03T05:32:58.488 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-06-03T05:32:58.749 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:32:58.749 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T05:32:58.749 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:32:58.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-06-03T05:32:59.381 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 4 to deep-scrub 2024-06-03T05:32:59.393 INFO:tasks.ceph:Scrubbing osd.5 2024-06-03T05:32:59.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-06-03T05:32:59.610 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:32:59.610 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T05:32:59.610 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:32:59.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-06-03T05:33:00.266 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 5 to deep-scrub 2024-06-03T05:33:00.280 INFO:tasks.ceph:Scrubbing osd.6 2024-06-03T05:33:00.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-06-03T05:33:00.485 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:33:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T05:33:00.486 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:33:00.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-06-03T05:33:01.176 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 6 to deep-scrub 2024-06-03T05:33:01.189 INFO:tasks.ceph:Scrubbing osd.7 2024-06-03T05:33:01.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-06-03T05:33:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-06-03T05:33:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T05:33:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-06-03T05:33:01.408 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-06-03T05:33:02.028 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 7 to deep-scrub 2024-06-03T05:33:02.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:33:02.661 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:33:02.662 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:33:02.673 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-06-03T05:33:02.400046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239436,"kb_used_data":8560,"kb_used_omap":20,"kb_used_meta":218091,"kb_avail":754735284,"statfs":{"total":773094113280,"available":772848930816,"internally_reserved":0,"allocated":8765440,"data_stored":4542424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21175,"internal_metadata":223325513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":154,"apply_latency_ms":154,"commit_latency_ns":154000000,"apply_latency_ns":154000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002143"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":232,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:33:00.232789+0000","last_change":"2024-06-03T05:33:00.233636+0000","last_active":"2024-06-03T05:33:00.232789+0000","last_peered":"2024-06-03T05:33:00.232789+0000","last_clean":"2024-06-03T05:33:00.232789+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:33:00.232789+0000","last_undegraded":"2024-06-03T05:33:00.232789+0000","last_fullsized":"2024-06-03T05:33:00.232789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:33:00.231131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:33:00.231131+0000","last_clean_scrub_stamp":"2024-06-03T05:33:00.231131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:14:30.902690+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":106,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:46.328148+0000","last_change":"2024-06-03T05:29:45.112876+0000","last_active":"2024-06-03T05:32:46.328148+0000","last_peered":"2024-06-03T05:32:46.328148+0000","last_clean":"2024-06-03T05:32:46.328148+0000","last_became_active":"2024-06-03T05:29:45.083288+0000","last_became_peered":"2024-06-03T05:29:45.083288+0000","last_unstale":"2024-06-03T05:32:46.328148+0000","last_undegraded":"2024-06-03T05:32:46.328148+0000","last_fullsized":"2024-06-03T05:32:46.328148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:28:04.376061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":231,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.631400+0000","last_change":"2024-06-03T05:29:45.181097+0000","last_active":"2024-06-03T05:32:47.631400+0000","last_peered":"2024-06-03T05:32:47.631400+0000","last_clean":"2024-06-03T05:32:47.631400+0000","last_became_active":"2024-06-03T05:29:45.174231+0000","last_became_peered":"2024-06-03T05:29:45.174231+0000","last_unstale":"2024-06-03T05:32:47.631400+0000","last_undegraded":"2024-06-03T05:32:47.631400+0000","last_fullsized":"2024-06-03T05:32:47.631400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.623013+0000","last_change":"2024-06-03T05:32:25.005092+0000","last_active":"2024-06-03T05:32:47.623013+0000","last_peered":"2024-06-03T05:32:47.623013+0000","last_clean":"2024-06-03T05:32:47.623013+0000","last_became_active":"2024-06-03T05:32:24.975211+0000","last_became_peered":"2024-06-03T05:32:24.975211+0000","last_unstale":"2024-06-03T05:32:47.623013+0000","last_undegraded":"2024-06-03T05:32:47.623013+0000","last_fullsized":"2024-06-03T05:32:47.623013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:20:31.052690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":230,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.624799+0000","last_change":"2024-06-03T05:28:54.917780+0000","last_active":"2024-06-03T05:32:47.624799+0000","last_peered":"2024-06-03T05:32:47.624799+0000","last_clean":"2024-06-03T05:32:47.624799+0000","last_became_active":"2024-06-03T05:28:54.908410+0000","last_became_peered":"2024-06-03T05:28:54.908410+0000","last_unstale":"2024-06-03T05:32:47.624799+0000","last_undegraded":"2024-06-03T05:32:47.624799+0000","last_fullsized":"2024-06-03T05:32:47.624799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":233,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.614380+0000","last_change":"2024-06-03T05:32:24.559256+0000","last_active":"2024-06-03T05:32:47.614380+0000","last_peered":"2024-06-03T05:32:47.614380+0000","last_clean":"2024-06-03T05:32:47.614380+0000","last_became_active":"2024-06-03T05:32:24.553004+0000","last_became_peered":"2024-06-03T05:32:24.553004+0000","last_unstale":"2024-06-03T05:32:47.614380+0000","last_undegraded":"2024-06-03T05:32:47.614380+0000","last_fullsized":"2024-06-03T05:32:47.614380+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:20:48.090948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":237,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:47.621582+0000","last_change":"2024-06-03T05:32:24.556287+0000","last_active":"2024-06-03T05:32:47.621582+0000","last_peered":"2024-06-03T05:32:47.621582+0000","last_clean":"2024-06-03T05:32:47.621582+0000","last_became_active":"2024-06-03T05:32:24.549787+0000","last_became_peered":"2024-06-03T05:32:24.549787+0000","last_unstale":"2024-06-03T05:32:47.621582+0000","last_undegraded":"2024-06-03T05:32:47.621582+0000","last_fullsized":"2024-06-03T05:32:47.621582+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:24:13.082657+0000","last_clean_scrub_stamp":"2024-06-03T05:24:13.082657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":232,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:59.445563+0000","last_change":"2024-06-03T05:32:59.451052+0000","last_active":"2024-06-03T05:32:59.445563+0000","last_peered":"2024-06-03T05:32:59.445563+0000","last_clean":"2024-06-03T05:32:59.445563+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:32:59.445563+0000","last_undegraded":"2024-06-03T05:32:59.445563+0000","last_fullsized":"2024-06-03T05:32:59.445563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:32:59.438783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:32:59.438783+0000","last_clean_scrub_stamp":"2024-06-03T05:32:59.438783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T06:35:08.889205+0000","scrub_duration":346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"128'78","reported_seq":263,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:57.779680+0000","last_change":"2024-06-03T05:32:57.786262+0000","last_active":"2024-06-03T05:32:57.779680+0000","last_peered":"2024-06-03T05:32:57.779680+0000","last_clean":"2024-06-03T05:32:57.779680+0000","last_became_active":"2024-06-03T05:32:24.556417+0000","last_became_peered":"2024-06-03T05:32:24.556417+0000","last_unstale":"2024-06-03T05:32:57.779680+0000","last_undegraded":"2024-06-03T05:32:57.779680+0000","last_fullsized":"2024-06-03T05:32:57.779680+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'78","last_scrub_stamp":"2024-06-03T05:32:57.775068+0000","last_deep_scrub":"128'78","last_deep_scrub_stamp":"2024-06-03T05:32:57.775068+0000","last_clean_scrub_stamp":"2024-06-03T05:32:57.775068+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:53:02.419624+0000","scrub_duration":827,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204224,"data_stored":1193520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378740,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":83,"seq":356482285609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27707,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28372686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28380,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343460,"statfs":{"total":96636764160,"available":96607703040,"internally_reserved":0,"allocated":1396736,"data_stored":866183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":79,"apply_latency_ms":79,"commit_latency_ns":79000000,"apply_latency_ns":79000000},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":71,"seq":304942678068,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28503763},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":108,"seq":463856467979,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1364,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1396736,"data_stored":866183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28176072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:33:02.677 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-06-03T05:24:13.082657+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=24, tm_sec=13, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=32, tm_sec=54, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T05:33:02.677 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-06-03T05:24:13.082657+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=24, tm_sec=13, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=32, tm_sec=54, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T05:33:02.677 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-06-03T05:24:13.082657+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=24, tm_sec=13, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=32, tm_sec=54, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T05:33:02.677 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-06-03T05:24:13.082657+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=24, tm_sec=13, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=32, tm_sec=54, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T05:33:02.677 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-06-03T05:24:13.082657+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=24, tm_sec=13, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=32, tm_sec=54, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T05:33:02.677 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-06-03T05:24:13.082657+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=24, tm_sec=13, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=5, tm_min=32, tm_sec=54, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T05:33:02.678 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-06-03T05:33:10.542 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:05:27.184 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:10.549 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:05:27.191 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:10.568 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:05:27.210 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:10.597 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:03:14.408 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:10.598 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:04:23.876 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:10.602 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:04:23.881 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:10.604 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:03:14.415 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:10.618 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:04:23.896 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:10.623 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:03:14.434 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:22.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T05:33:23.275 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-06-03T05:33:23.275 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-06-03T05:33:23.287 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-06-03T05:33:22.404891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239412,"kb_used_data":8560,"kb_used_omap":20,"kb_used_meta":218091,"kb_avail":754735308,"statfs":{"total":773094113280,"available":772848955392,"internally_reserved":0,"allocated":8765440,"data_stored":4542424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21175,"internal_metadata":223325513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002979"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":232,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:33:00.232789+0000","last_change":"2024-06-03T05:33:00.233636+0000","last_active":"2024-06-03T05:33:00.232789+0000","last_peered":"2024-06-03T05:33:00.232789+0000","last_clean":"2024-06-03T05:33:00.232789+0000","last_became_active":"2024-06-03T05:24:15.491548+0000","last_became_peered":"2024-06-03T05:24:15.491548+0000","last_unstale":"2024-06-03T05:33:00.232789+0000","last_undegraded":"2024-06-03T05:33:00.232789+0000","last_fullsized":"2024-06-03T05:33:00.232789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:33:00.231131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:33:00.231131+0000","last_clean_scrub_stamp":"2024-06-03T05:33:00.231131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:14:30.902690+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":115,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:33:00.328308+0000","last_change":"2024-06-03T05:33:00.338530+0000","last_active":"2024-06-03T05:33:00.328308+0000","last_peered":"2024-06-03T05:33:00.328308+0000","last_clean":"2024-06-03T05:33:00.328308+0000","last_became_active":"2024-06-03T05:29:45.083288+0000","last_became_peered":"2024-06-03T05:29:45.083288+0000","last_unstale":"2024-06-03T05:33:00.328308+0000","last_undegraded":"2024-06-03T05:33:00.328308+0000","last_fullsized":"2024-06-03T05:33:00.328308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:33:00.323716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:33:00.323716+0000","last_clean_scrub_stamp":"2024-06-03T05:33:00.323716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T05:33:03.074375+0000","scrub_duration":398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":239,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:33:02.194151+0000","last_change":"2024-06-03T05:33:02.195008+0000","last_active":"2024-06-03T05:33:02.194151+0000","last_peered":"2024-06-03T05:33:02.194151+0000","last_clean":"2024-06-03T05:33:02.194151+0000","last_became_active":"2024-06-03T05:29:45.174231+0000","last_became_peered":"2024-06-03T05:29:45.174231+0000","last_unstale":"2024-06-03T05:33:02.194151+0000","last_undegraded":"2024-06-03T05:33:02.194151+0000","last_fullsized":"2024-06-03T05:33:02.194151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:33:02.192360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:33:02.192360+0000","last_clean_scrub_stamp":"2024-06-03T05:33:02.192360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T06:16:45.532266+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":54,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:33:00.737924+0000","last_change":"2024-06-03T05:33:00.743957+0000","last_active":"2024-06-03T05:33:00.737924+0000","last_peered":"2024-06-03T05:33:00.737924+0000","last_clean":"2024-06-03T05:33:00.737924+0000","last_became_active":"2024-06-03T05:32:24.975211+0000","last_became_peered":"2024-06-03T05:32:24.975211+0000","last_unstale":"2024-06-03T05:33:00.737924+0000","last_undegraded":"2024-06-03T05:33:00.737924+0000","last_fullsized":"2024-06-03T05:33:00.737924+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:33:00.733538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:33:00.733538+0000","last_clean_scrub_stamp":"2024-06-03T05:33:00.733538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:48:15.139830+0000","scrub_duration":289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":238,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:33:01.170881+0000","last_change":"2024-06-03T05:33:01.172141+0000","last_active":"2024-06-03T05:33:01.170881+0000","last_peered":"2024-06-03T05:33:01.170881+0000","last_clean":"2024-06-03T05:33:01.170881+0000","last_became_active":"2024-06-03T05:28:54.908410+0000","last_became_peered":"2024-06-03T05:28:54.908410+0000","last_unstale":"2024-06-03T05:33:01.170881+0000","last_undegraded":"2024-06-03T05:33:01.170881+0000","last_fullsized":"2024-06-03T05:33:01.170881+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:33:01.169189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:33:01.169189+0000","last_clean_scrub_stamp":"2024-06-03T05:33:01.169189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T05:55:55.873702+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":242,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:33:03.227069+0000","last_change":"2024-06-03T05:33:03.232455+0000","last_active":"2024-06-03T05:33:03.227069+0000","last_peered":"2024-06-03T05:33:03.227069+0000","last_clean":"2024-06-03T05:33:03.227069+0000","last_became_active":"2024-06-03T05:32:24.553004+0000","last_became_peered":"2024-06-03T05:32:24.553004+0000","last_unstale":"2024-06-03T05:33:03.227069+0000","last_undegraded":"2024-06-03T05:33:03.227069+0000","last_fullsized":"2024-06-03T05:33:03.227069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:33:03.222466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:33:03.222466+0000","last_clean_scrub_stamp":"2024-06-03T05:33:03.222466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:07:54.219579+0000","scrub_duration":244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":245,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:33:03.144048+0000","last_change":"2024-06-03T05:33:03.144917+0000","last_active":"2024-06-03T05:33:03.144048+0000","last_peered":"2024-06-03T05:33:03.144048+0000","last_clean":"2024-06-03T05:33:03.144048+0000","last_became_active":"2024-06-03T05:32:24.549787+0000","last_became_peered":"2024-06-03T05:32:24.549787+0000","last_unstale":"2024-06-03T05:33:03.144048+0000","last_undegraded":"2024-06-03T05:33:03.144048+0000","last_fullsized":"2024-06-03T05:33:03.144048+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:33:03.139816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:33:03.139816+0000","last_clean_scrub_stamp":"2024-06-03T05:33:03.139816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:14:26.407877+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":232,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:59.445563+0000","last_change":"2024-06-03T05:32:59.451052+0000","last_active":"2024-06-03T05:32:59.445563+0000","last_peered":"2024-06-03T05:32:59.445563+0000","last_clean":"2024-06-03T05:32:59.445563+0000","last_became_active":"2024-06-03T05:24:15.011995+0000","last_became_peered":"2024-06-03T05:24:15.011995+0000","last_unstale":"2024-06-03T05:32:59.445563+0000","last_undegraded":"2024-06-03T05:32:59.445563+0000","last_fullsized":"2024-06-03T05:32:59.445563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T05:32:59.438783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T05:32:59.438783+0000","last_clean_scrub_stamp":"2024-06-03T05:32:59.438783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T06:35:08.889205+0000","scrub_duration":346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"128'78","reported_seq":263,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-06-03T05:32:57.779680+0000","last_change":"2024-06-03T05:32:57.786262+0000","last_active":"2024-06-03T05:32:57.779680+0000","last_peered":"2024-06-03T05:32:57.779680+0000","last_clean":"2024-06-03T05:32:57.779680+0000","last_became_active":"2024-06-03T05:32:24.556417+0000","last_became_peered":"2024-06-03T05:32:24.556417+0000","last_unstale":"2024-06-03T05:32:57.779680+0000","last_undegraded":"2024-06-03T05:32:57.779680+0000","last_fullsized":"2024-06-03T05:32:57.779680+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'78","last_scrub_stamp":"2024-06-03T05:32:57.775068+0000","last_deep_scrub":"128'78","last_deep_scrub_stamp":"2024-06-03T05:32:57.775068+0000","last_clean_scrub_stamp":"2024-06-03T05:32:57.775068+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:53:02.419624+0000","scrub_duration":827,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1193520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204224,"data_stored":1193520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":18,"seq":77309411445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":83,"seq":356482285613,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27707,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28372686},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":18,"seq":77309411448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28380,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343460,"statfs":{"total":96636764160,"available":96607703040,"internally_reserved":0,"allocated":1396736,"data_stored":866183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":16,"seq":68719476854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":71,"seq":304942678072,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":972,"kb_used_omap":4,"kb_used_meta":27835,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":995328,"data_stored":468343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4397,"internal_metadata":28503763},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":108,"seq":463856467983,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1364,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1396736,"data_stored":866183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4408,"internal_metadata":28176072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T05:33:23.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-06-03T05:33:24.106 INFO:teuthology.misc:Shutting down mds daemons... 2024-06-03T05:33:24.107 INFO:teuthology.misc:Shutting down osd daemons... 2024-06-03T05:33:24.107 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-06-03T05:33:24.107 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:33:24.116 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:33:24.107+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35324) UID: 0 2024-06-03T05:33:24.116 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-06-03T05:33:24.108+0000 b485640 -1 osd.0 128 *** Got signal Terminated *** 2024-06-03T05:33:30.758 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:05:47.399 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:30.764 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:05:47.406 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:30.782 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:05:47.424 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:30.812 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:04:44.091 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:30.814 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:04:44.092 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:30.829 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:04:44.107 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:30.831 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:03:34.642 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:30.835 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:03:34.645 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:30.852 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:03:34.663 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:50.693 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:07.334 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:50.694 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:07.336 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:50.706 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:07.349 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:50.742 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:03:54.553 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:50.744 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:04.022 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:50.745 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:03:54.556 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:50.745 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:04.024 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:50.759 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:04.037 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:50.762 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:03:54.573 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:33:59.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-06-03T05:34:00.715 INFO:tasks.ceph.osd.0:Stopped 2024-06-03T05:34:00.716 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-06-03T05:34:00.716 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:34:00.724 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:34:00.717+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35332) UID: 0 2024-06-03T05:34:00.725 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-06-03T05:34:00.718+0000 b485640 -1 osd.1 132 *** Got signal Terminated *** 2024-06-03T05:34:05.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-06-03T05:34:05.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-06-03T05:34:06.818 INFO:tasks.ceph.osd.1:Stopped 2024-06-03T05:34:06.819 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-06-03T05:34:06.819 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:34:06.827 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:34:06.819+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35340) UID: 0 2024-06-03T05:34:06.827 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-06-03T05:34:06.819+0000 b485640 -1 osd.2 134 *** Got signal Terminated *** 2024-06-03T05:34:11.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-06-03T05:34:11.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-06-03T05:34:12.921 INFO:tasks.ceph.osd.2:Stopped 2024-06-03T05:34:12.921 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-06-03T05:34:12.921 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:34:12.930 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:34:12.922+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35341) UID: 0 2024-06-03T05:34:12.930 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-06-03T05:34:12.922+0000 b485640 -1 osd.3 136 *** Got signal Terminated *** 2024-06-03T05:34:15.800 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:32.442 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:15.807 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:29.085 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:15.808 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:19.619 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:15.849 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:32.491 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:15.854 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:29.132 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:15.856 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:19.667 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:15.875 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:29.154 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:15.877 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:32.519 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:15.879 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:19.690 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:17.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-06-03T05:34:17.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-06-03T05:34:19.023 INFO:tasks.ceph.osd.3:Stopped 2024-06-03T05:34:19.024 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-06-03T05:34:19.024 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:34:19.033 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:34:19.024+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 58888) UID: 0 2024-06-03T05:34:19.033 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-06-03T05:34:19.024+0000 b485640 -1 osd.4 140 *** Got signal Terminated *** 2024-06-03T05:34:23.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-06-03T05:34:34.744 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:51.386 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:34.749 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:51.391 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:34.769 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:51.411 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:34.802 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:48.080 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:34.804 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:48.082 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:34.804 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:38.615 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:34.805 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:38.616 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:34.825 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:48.104 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:34.827 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:38.637 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:35.877 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:52.519 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:35.880 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:52.522 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:35.889 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:06:52.531 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:35.928 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:49.206 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:35.929 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:49.208 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:35.938 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:05:49.216 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:35.949 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:39.760 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:35.950 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:39.761 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:35.960 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:39.770 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:55.927 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:07:12.567 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:55.931 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:07:12.573 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:55.946 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:07:12.588 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:55.979 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:06:09.257 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:55.980 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:06:09.259 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:55.994 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:06:09.272 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:56.000 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:59.810 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:56.001 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:59.812 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:34:56.016 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:04:59.826 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:16.003 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:07:32.645 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:16.029 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:07:32.671 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:16.042 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:06:29.320 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:16.044 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:06:29.323 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:16.048 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:07:32.690 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:16.058 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:06:29.336 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:16.061 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:05:19.872 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:16.063 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:05:19.874 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:16.086 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:05:19.897 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:36.078 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:07:52.720 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:36.079 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:07:52.722 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:36.095 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:07:52.737 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:36.123 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:06:49.401 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:36.126 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:06:49.404 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:36.131 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:05:39.941 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:36.132 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:05:39.942 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:36.139 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:06:49.418 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:36.146 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:05:39.956 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:50.547 INFO:tasks.ceph.osd.4:Stopped 2024-06-03T05:35:50.548 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-06-03T05:35:50.548 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:35:50.562 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:35:50.552+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 64840) UID: 0 2024-06-03T05:35:50.562 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-06-03T05:35:50.554+0000 b485640 -1 osd.5 142 *** Got signal Terminated *** 2024-06-03T05:35:56.124 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:08:12.766 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:56.128 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:08:12.770 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:56.151 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:08:12.793 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:56.180 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:07:09.459 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:56.182 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:07:09.460 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:56.187 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:05:59.997 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:56.188 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:05:59.998 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:56.198 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:07:09.476 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:56.204 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:06:00.015 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:35:56.649 INFO:tasks.ceph.osd.5:Stopped 2024-06-03T05:35:56.650 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-06-03T05:35:56.650 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:35:56.658 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:35:56.651+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 55322) UID: 0 2024-06-03T05:35:56.658 INFO:tasks.ceph.osd.6.smithi064.stderr:2024-06-03T05:35:56.651+0000 b485640 -1 osd.6 144 *** Got signal Terminated *** 2024-06-03T05:35:57.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-06-03T05:36:02.751 INFO:tasks.ceph.osd.6:Stopped 2024-06-03T05:36:02.751 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-06-03T05:36:02.751 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:36:02.760 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:36:02.752+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35180) UID: 0 2024-06-03T05:36:02.760 INFO:tasks.ceph.osd.7.smithi064.stderr:2024-06-03T05:36:02.752+0000 b485640 -1 osd.7 144 *** Got signal Terminated *** 2024-06-03T05:36:02.831 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-06-03T05:36:02.831 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-06-03T05:36:08.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-06-03T05:36:16.175 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:08:32.817 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:16.178 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:08:32.820 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:16.192 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:08:32.834 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:16.227 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:07:29.506 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:16.229 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:07:29.507 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:16.230 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:06:20.041 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:16.232 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:06:20.043 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:16.242 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:07:29.521 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:16.253 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:06:20.064 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:36.223 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:08:52.865 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:36.227 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:08:52.869 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:36.243 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:08:52.885 59018== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:36.277 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:07:49.555 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:36.278 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:07:49.556 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:36.280 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:06:40.091 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:36.283 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:06:40.094 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:36.292 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:07:49.570 57894== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:36.305 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:06:40.116 74502== Warning: unimplemented fcntl command: 1036 2024-06-03T05:36:45.462 INFO:tasks.ceph.osd.7:Stopped 2024-06-03T05:36:45.462 INFO:teuthology.misc:Shutting down mgr daemons... 2024-06-03T05:36:45.463 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-06-03T05:36:45.463 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:36:45.493 INFO:tasks.ceph.mgr.y:Stopped 2024-06-03T05:36:45.493 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-06-03T05:36:45.493 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:36:45.521 INFO:tasks.ceph.mgr.x:Stopped 2024-06-03T05:36:45.521 INFO:teuthology.misc:Shutting down mon daemons... 2024-06-03T05:36:45.522 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-03T05:36:45.522 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:36:45.549 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-06-03T05:36:45.538+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 58968) UID: 0 2024-06-03T05:36:45.550 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-06-03T05:36:45.540+0000 9f9e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-03T05:36:46.309 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:09:02.951 59018== 2024-06-03T05:36:46.309 INFO:tasks.ceph.mon.a.smithi038.stderr:==00:00:09:02.951 59018== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:36:46.322 INFO:tasks.ceph.mon.a.smithi038.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:36:47.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-06-03T05:36:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:36:51.623 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-06-03T05:36:51.624 INFO:tasks.ceph.mon.a:Stopped 2024-06-03T05:36:51.624 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-03T05:36:51.624 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:36:51.641 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:36:51.628+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 74497) UID: 0 2024-06-03T05:36:51.641 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-06-03T05:36:51.630+0000 9f9e640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-06-03T05:36:52.392 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:06:56.202 74502== 2024-06-03T05:36:52.392 INFO:tasks.ceph.mon.c.smithi038.stderr:==00:00:06:56.202 74502== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:36:52.405 INFO:tasks.ceph.mon.c.smithi038.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:36:52.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-06-03T05:36:57.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-06-03T05:36:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:36:57.726 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-06-03T05:36:57.726 INFO:tasks.ceph.mon.c:Stopped 2024-06-03T05:36:57.727 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-03T05:36:57.727 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T05:36:57.743 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:36:57.730+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 57892) UID: 0 2024-06-03T05:36:57.743 INFO:tasks.ceph.mon.b.smithi064.stderr:2024-06-03T05:36:57.732+0000 9f9e640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-06-03T05:37:02.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-06-03T05:41:47.275 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:13:00.553 57894== 2024-06-03T05:41:47.275 INFO:tasks.ceph.mon.b.smithi064.stderr:==00:00:13:00.553 57894== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T05:41:47.285 INFO:tasks.ceph.mon.b.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-06-03T05:41:48.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-06-03T05:41:50.591 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T05:41:50.591 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-06-03T05:41:50.592 INFO:tasks.ceph.mon.b:Stopped 2024-06-03T05:41:50.592 INFO:tasks.ceph:Checking cluster log for badness... 2024-06-03T05:41:50.593 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(SLOW OPS\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-06-03T05:41:50.637 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:41:50.637 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-06-03T05:41:50.733 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:41:50.733 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-06-03T05:41:50.806 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:41:50.806 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-06-03T05:41:50.858 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:41:50.858 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-06-03T05:41:50.914 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:41:50.915 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-06-03T05:41:50.989 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:41:50.989 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-06-03T05:41:51.178 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:41:51.178 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-06-03T05:41:51.243 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:41:51.243 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-06-03T05:41:51.320 INFO:tasks.ceph:Archiving mon data... 2024-06-03T05:41:51.322 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/data/mon.a.tgz 2024-06-03T05:41:51.322 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-06-03T05:41:51.339 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.I8uun5WWTl 2024-06-03T05:41:51.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.I8uun5WWTl 2024-06-03T05:41:51.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.I8uun5WWTl 2024-06-03T05:41:51.569 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.I8uun5WWTl is 317KB 2024-06-03T05:41:51.665 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.I8uun5WWTl 2024-06-03T05:41:51.681 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/data/mon.c.tgz 2024-06-03T05:41:51.681 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-06-03T05:41:51.737 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.FRpWzVl4Nd 2024-06-03T05:41:51.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.FRpWzVl4Nd 2024-06-03T05:41:51.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.FRpWzVl4Nd 2024-06-03T05:41:51.945 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.FRpWzVl4Nd is 317KB 2024-06-03T05:41:52.013 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.FRpWzVl4Nd 2024-06-03T05:41:52.029 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/data/mon.b.tgz 2024-06-03T05:41:52.030 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-06-03T05:41:52.049 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.qMtsoTjYxt 2024-06-03T05:41:52.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.qMtsoTjYxt 2024-06-03T05:41:52.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.qMtsoTjYxt 2024-06-03T05:41:52.278 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.qMtsoTjYxt is 317KB 2024-06-03T05:41:52.374 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.qMtsoTjYxt 2024-06-03T05:41:52.390 INFO:tasks.ceph:Cleaning ceph cluster... 2024-06-03T05:41:52.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-03T05:41:52.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-03T05:41:52.721 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-06-03T05:41:52.994 INFO:tasks.ceph:Archiving crash dumps... 2024-06-03T05:41:52.996 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/remote/smithi038/crash 2024-06-03T05:41:52.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-03T05:41:53.023 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/remote/smithi064/crash 2024-06-03T05:41:53.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-03T05:41:53.051 INFO:tasks.ceph:Compressing logs... 2024-06-03T05:41:53.051 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-03T05:41:53.068 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-03T05:41:53.168 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.44681.log 2024-06-03T05:41:53.168 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34785.log 2024-06-03T05:41:53.168 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.44681.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.64060.log 2024-06-03T05:41:53.168 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34785.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34785.log.gz 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.64060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35083.log 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.45484.log 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35083.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35083.log.gz 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/objectstore_tool.48781.log 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.45484.log: /var/log/ceph/objectstore_tool.48781.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34923.log 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34840.log 2024-06-03T05:41:53.169 INFO:teuthology.orchestra.run.smithi064.stderr: 88.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34923.log.gz 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34840.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34840.log.gz 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.5.log: -- /var/log/ceph/ceph-client.admin.35043.log 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35043.log.gz 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr: 94.7% -- replaced with /var/log/ceph/objectstore_tool.44681.log.gz 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mgr.x.log: 94.8% -- replaced with /var/log/ceph/objectstore_tool.64060.log.gzgzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-06-03T05:41:53.170 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/mon.b.log: 69.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr: 64.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 94.7% -- replaced with /var/log/ceph/objectstore_tool.48781.log.gz 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr: 94.3% -- replaced with /var/log/ceph/objectstore_tool.45484.log.gz 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.47111.log 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr: 64.1% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.47111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35003.log 2024-06-03T05:41:53.171 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.42975.log 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35003.log.gzgzip 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/objectstore_tool.43838.log 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34883.log 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.43838.log: /var/log/ceph/objectstore_tool.42975.log: 94.7% -- replaced with /var/log/ceph/objectstore_tool.47111.log.gz 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34963.log 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34883.log.gz 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.50439.log 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr: 94.7% -- replaced with /var/log/ceph/objectstore_tool.43838.log.gz 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34963.log.gz 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.48012.log 2024-06-03T05:41:53.172 INFO:teuthology.orchestra.run.smithi064.stderr: 94.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-06-03T05:41:53.173 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.50439.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.49676.log 2024-06-03T05:41:53.173 INFO:teuthology.orchestra.run.smithi064.stderr: 94.7% -- replaced with /var/log/ceph/objectstore_tool.42975.log.gz 2024-06-03T05:41:53.173 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.48012.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-06-03T05:41:53.173 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.49676.log: /var/log/ceph/ceph-mon.b.log: 94.5% 94.6% -- replaced with /var/log/ceph/objectstore_tool.50439.log.gz 2024-06-03T05:41:53.173 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/objectstore_tool.48012.log.gz 2024-06-03T05:41:53.173 INFO:teuthology.orchestra.run.smithi064.stderr: 94.7% -- replaced with /var/log/ceph/objectstore_tool.49676.log.gz 2024-06-03T05:41:53.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-06-03T05:41:53.263 INFO:teuthology.orchestra.run.smithi064.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-06-03T05:41:53.275 INFO:teuthology.orchestra.run.smithi064.stderr: 90.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-06-03T05:41:53.295 INFO:teuthology.orchestra.run.smithi064.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-06-03T05:41:53.315 INFO:teuthology.orchestra.run.smithi064.stderr: 90.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-06-03T05:41:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67101.log 2024-06-03T05:41:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73115.log 2024-06-03T05:41:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.67101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74411.log 2024-06-03T05:41:53.334 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67101.log.gz 2024-06-03T05:41:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40910.log 2024-06-03T05:41:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73115.log.gz 2024-06-03T05:41:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74564.log 2024-06-03T05:41:53.335 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74411.log.gz 2024-06-03T05:41:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2024-06-03T05:41:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44303.log 2024-06-03T05:41:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44243.log 2024-06-03T05:41:53.335 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74564.log.gz 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70417.log 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44303.log.gz 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78727.log 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44243.log.gz 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79154.log 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70417.log.gz 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47052.log 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78727.log.gz 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46708.log 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79154.log.gz 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40135.log 2024-06-03T05:41:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47052.log.gz 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76167.log 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46708.log.gz 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39561.log 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40135.log.gz 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79193.log 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.76167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76167.log.gz 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65459.log 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39561.log.gz 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76271.log 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79193.log.gz 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75311.log 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65459.log.gz 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.76271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78530.log 2024-06-03T05:41:53.337 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76271.log.gz 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.75311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42402.log 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75311.log.gz 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42922.log 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78530.log.gz 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77898.log 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42402.log.gz 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43977.log 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42922.log.gz 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47113.log 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77898.log.gz 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43344.log 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43977.log.gz 2024-06-03T05:41:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72658.log 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47113.log.gz 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50741.log 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43344.log.gz 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66692.log 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72658.log.gz 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40725.log 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50741.log.gz 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72492.log 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66692.log.gz 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60127.log 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40725.log.gz 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72198.log 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72492.log.gz 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.60127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78534.log 2024-06-03T05:41:53.339 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60127.log.gz 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78541.log 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72198.log.gz 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61782.log 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78534.log.gz 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41387.log 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78541.log.gz 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.61782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80040.log 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61782.log.gz 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43503.log 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41387.log.gz 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59585.log 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80040.log.gz 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/ceph-client.admin.43503.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51996.log 2024-06-03T05:41:53.340 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43503.log.gz 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55432.log 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59585.log.gz 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62275.log 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.51996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51996.log.gz 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79294.log 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55432.log.gz 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69273.log 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62275.log.gz 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66626.log 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79294.log.gz 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.69273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43184.log 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69273.log.gz 2024-06-03T05:41:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71429.log 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66626.log.gz 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39777.log 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43184.log.gz 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46794.log 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71429.log.gz 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39777.log.gz 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46794.log.gz 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73561.log 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2024-06-03T05:41:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77688.log 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73601.log 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73561.log.gz 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74339.log 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77688.log.gz 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75072.log 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73601.log.gz 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78360.log 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74339.log.gz 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.75072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54793.log 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75072.log.gz 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44866.log 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78360.log.gz 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42512.log 2024-06-03T05:41:53.343 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54793.log.gz 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39544.log 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44866.log.gz 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr: 93.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50363.log 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42512.log.gz 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63057.log 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39544.log.gz 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50107.log 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50363.log.gz 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78908.log 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63057.log.gz 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50107.log.gz 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77730.log 2024-06-03T05:41:53.344 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78908.log.gz 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78540.log 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63279.log 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77730.log.gz 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.78540.log: /var/log/ceph/ceph-client.admin.63477.log 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78540.log.gz 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66865.log 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63279.log.gz 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67745.log 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63477.log.gz 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64439.log 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66865.log.gz 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.67745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73644.log 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67745.log.gz 2024-06-03T05:41:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64439.log.gz 2024-06-03T05:41:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78542.log 2024-06-03T05:41:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5/var/log/ceph/ceph-client.admin.73644.log: --verbose -- /var/log/ceph/ceph-client.admin.73011.log 2024-06-03T05:41:53.346 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73644.log.gz 2024-06-03T05:41:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72812.log 2024-06-03T05:41:53.346 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78542.log.gz 2024-06-03T05:41:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59986.log 2024-06-03T05:41:53.346 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73011.log.gz 2024-06-03T05:41:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47968.log 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72812.log.gz 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77856.log 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59986.log.gz 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49006.log 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47968.log.gz 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79662.log 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77856.log.gz 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35350.log 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49006.log.gz 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66525.log 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79662.log.gz 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62635.log 2024-06-03T05:41:53.347 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35350.log.gz 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66525.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66525.log.gz 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62079.log 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62635.log.gz 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40599.log 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71606.log 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62079.log.gz 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79336.log 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40599.log.gz 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71606.log.gz 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80082.log 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59482.log 2024-06-03T05:41:53.348 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79336.log.gz 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43895.log 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80082.log.gz 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59482.log.gz 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44526.log 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43895.log.gz 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41440.log 0.0% 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72409.log 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44526.log.gz 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41440.log.gz 2024-06-03T05:41:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63699.log 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55148.log/var/log/ceph/ceph-client.admin.72409.log: 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72409.log.gz 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75729.log 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63699.log.gz 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55148.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.55148.log.gz 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39562.log 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.75729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76568.log 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75729.log.gz 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66063.log 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39562.log.gz 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.76568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73991.log 2024-06-03T05:41:53.350 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76568.log.gz 2024-06-03T05:41:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57343.log 2024-06-03T05:41:53.351 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66063.log.gz 2024-06-03T05:41:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65742.log 2024-06-03T05:41:53.351 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73991.log.gz 2024-06-03T05:41:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.57343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68799.log 2024-06-03T05:41:53.351 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57343.log.gz 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48495.log 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65742.log.gz 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.68799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78905.log 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68799.log.gz 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63157.log 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48495.log.gz 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70097.log 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78905.log.gz 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49448.log 2024-06-03T05:41:53.352 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63157.log.gz 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70097.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.70097.log.gz 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/ceph-client.admin.46072.log 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37344.log 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49448.log.gz 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46072.log.gz 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74708.log 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37344.log.gz 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72220.log 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74708.log.gz 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70501.log 2024-06-03T05:41:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43064.log 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72220.log.gz 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39814.log 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70501.log.gz 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44612.log 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43064.log.gz 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56062.log 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39814.log.gz 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72914.log 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44612.log.gz 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79494.log 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56062.log.gz 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74808.log 2024-06-03T05:41:53.354 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72914.log.gz 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66252.log 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79494.log.gz 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74195.log 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74808.log.gz 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64359.log 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66252.log.gz 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62675.log 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74195.log.gz 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64359.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29175.log 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64359.log.gz 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65037.log 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62675.log.gz 2024-06-03T05:41:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.tmp-client.admin.29175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64500.log 2024-06-03T05:41:53.356 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29175.log.gz 2024-06-03T05:41:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65037.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.65037.log.gz 2024-06-03T05:41:53.356 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/ceph-client.admin.59878.log 2024-06-03T05:41:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43947.log 2024-06-03T05:41:53.356 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64500.log.gz 2024-06-03T05:41:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39819.log 2024-06-03T05:41:53.357 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59878.log.gz 2024-06-03T05:41:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79578.log 2024-06-03T05:41:53.357 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43947.log.gz 2024-06-03T05:41:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78834.log 2024-06-03T05:41:53.357 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39819.log.gz 2024-06-03T05:41:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62214.log 2024-06-03T05:41:53.358 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79578.log.gz 2024-06-03T05:41:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40095.log 2024-06-03T05:41:53.358 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78834.log.gz 2024-06-03T05:41:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64997.log 2024-06-03T05:41:53.358 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62214.log.gz 2024-06-03T05:41:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73052.log 2024-06-03T05:41:53.359 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40095.log.gz 2024-06-03T05:41:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76748.log 2024-06-03T05:41:53.359 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64997.log.gz 2024-06-03T05:41:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73052.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.73052.log.gz -- 2024-06-03T05:41:53.359 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/ceph-client.admin.40431.log 2024-06-03T05:41:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.76748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60698.log 2024-06-03T05:41:53.360 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76748.log.gz 2024-06-03T05:41:53.360 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76468.log 2024-06-03T05:41:53.360 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2024-06-03T05:41:53.360 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.60698.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-06-03T05:41:53.360 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60698.log.gz 2024-06-03T05:41:53.360 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.76468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74340.log 2024-06-03T05:41:53.361 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76468.log.gz 2024-06-03T05:41:53.361 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66323.log 2024-06-03T05:41:53.361 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74340.log.gz 2024-06-03T05:41:53.361 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-06-03T05:41:53.361 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78833.log 2024-06-03T05:41:53.361 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66323.log.gz 2024-06-03T05:41:53.361 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41357.log 2024-06-03T05:41:53.361 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67480.log 2024-06-03T05:41:53.362 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78833.log.gz 2024-06-03T05:41:53.362 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41560.log 2024-06-03T05:41:53.362 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41357.log.gz 2024-06-03T05:41:53.362 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.67480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76970.log 2024-06-03T05:41:53.362 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67480.log.gz 2024-06-03T05:41:53.362 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67960.log 2024-06-03T05:41:53.362 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41560.log.gz 2024-06-03T05:41:53.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.76970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71726.log 2024-06-03T05:41:53.363 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76970.log.gz 2024-06-03T05:41:53.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.67960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59145.log 2024-06-03T05:41:53.363 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67960.log.gz 2024-06-03T05:41:53.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77472.log 2024-06-03T05:41:53.363 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71726.log.gz 2024-06-03T05:41:53.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54731.log 2024-06-03T05:41:53.363 INFO:teuthology.orchestra.run.smithi038.stderr: 28.6% -- replaced with /var/log/ceph/ceph-client.admin.59145.log.gz 2024-06-03T05:41:53.364 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79194.log 2024-06-03T05:41:53.364 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77472.log.gz 2024-06-03T05:41:53.364 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79746.log 2024-06-03T05:41:53.364 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54731.log.gz 2024-06-03T05:41:53.364 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78234.log 2024-06-03T05:41:53.364 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79194.log.gz 2024-06-03T05:41:53.364 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79704.log 2024-06-03T05:41:53.364 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79746.log.gz 2024-06-03T05:41:53.365 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42962.log 2024-06-03T05:41:53.365 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78234.log.gz 2024-06-03T05:41:53.365 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59102.log 2024-06-03T05:41:53.365 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79704.log.gz 2024-06-03T05:41:53.365 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80542.log 2024-06-03T05:41:53.365 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42962.log.gz 2024-06-03T05:41:53.365 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40641.log 2024-06-03T05:41:53.365 INFO:teuthology.orchestra.run.smithi038.stderr: 27.5% -- replaced with /var/log/ceph/ceph-client.admin.59102.log.gz 2024-06-03T05:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59649.log 2024-06-03T05:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80542.log.gz 2024-06-03T05:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40641.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-06-03T05:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40641.log.gz 2024-06-03T05:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55639.log 2024-06-03T05:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59649.log.gz 2024-06-03T05:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42138.log 2024-06-03T05:41:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55639.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55639.log.gz 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.79788.log 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42138.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55545.log 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42138.log.gz 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55169.log 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79788.log.gz 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.55545.log: -- /var/log/ceph/ceph-client.admin.75394.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55545.log.gz 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55169.log.gz 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65439.log 2024-06-03T05:41:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.75394.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75394.log.gz 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.80376.log 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65439.log.gz 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46507.log 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80376.log.gz 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35451.log 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46507.log.gz 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44408.log 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35451.log.gz 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67243.log 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44408.log.gz 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80292.log 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.67243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67243.log.gz 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79956.log 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80292.log.gz 2024-06-03T05:41:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65282.log 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79956.log.gz 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40787.log 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65282.log.gz 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80124.log 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40787.log.gz 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66965.log 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80124.log.gz 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79094.log 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66965.log.gz 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48453.log 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79094.log.gz 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54580.log 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48453.log.gz 2024-06-03T05:41:53.369 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79074.log 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54580.log.gz 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73778.log 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79074.log.gz 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50505.log 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73778.log.gz 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72452.log 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50505.log.gz 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79830.log 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72452.log.gz 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79872.log 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79830.log.gz 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-06-03T05:41:53.370 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79872.log.gz 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44695.log 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80250.log 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44695.log: 0.0%gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44695.log.gz 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80250.log.gz 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/valgrind/mon.a.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-06-03T05:41:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/valgrind/osd.3.log: /var/log/ceph/valgrind/osd.1.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/valgrind/mon.c.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79998.log 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52796.log 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/valgrind/osd.2.log: 64.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79998.log.gz 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42809.log 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39215.log 2024-06-03T05:41:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52796.log.gz 2024-06-03T05:41:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42809.log.gz 2024-06-03T05:41:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78531.log 2024-06-03T05:41:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39215.log.gz 2024-06-03T05:41:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77772.log 2024-06-03T05:41:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76187.log 2024-06-03T05:41:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78531.log.gz 2024-06-03T05:41:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77772.log.gz 2024-06-03T05:41:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59019.log 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.76187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76187.log.gz 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72960.log 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59019.log.gz 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78066.log 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72960.log.gz 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35575.log 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78066.log.gz 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60536.log 2024-06-03T05:41:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35575.log.gz 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43545.log 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.60536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60536.log.gz 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39255.log 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43545.log.gz 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64081.log 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39255.log.gz 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70840.log 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64081.log.gz 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70881.log 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70840.log.gz 2024-06-03T05:41:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61038.log 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70881.log.gz 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73831.log 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.61038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61038.log.gz 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73340.log 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73831.log.gz 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73238.log 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73340.log.gz 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66465.log 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73238.log.gz 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50565.log 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66465.log.gz 2024-06-03T05:41:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59216.log 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50565.log.gz 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62498.log 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59216.log.gz 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46029.log 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62498.log.gz 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39555.log 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46029.log.gz 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73686.log 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39555.log.gz 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79410.log 2024-06-03T05:41:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73686.log.gz 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62543.log 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79410.log.gz 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43565.log 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62543.log.gz 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80418.log 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43565.log.gz 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79450.log 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80418.log.gz 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63414.log 2024-06-03T05:41:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79450.log.gz 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54351.log 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63414.log.gz 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77514.log 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54351.log.gz 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39547.log 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77514.log.gz 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40906.log 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39547.log.gz 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44906.log 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40906.log.gz 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70927.log 2024-06-03T05:41:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44906.log.gz 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43264.log 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70927.log.gz 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80460.log 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43264.log.gz 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79620.log 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80460.log.gz 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39899.log 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79620.log.gz 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78753.log 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39899.log.gz 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55922.log 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78753.log.gz 2024-06-03T05:41:53.380 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80208.log 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55922.log.gz 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39381.log 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80208.log.gz 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65961.log 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39381.log.gz 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40473.log 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65961.log.gz 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69058.log 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40473.log.gz 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47292.log 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.69058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69058.log.gz 2024-06-03T05:41:53.381 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38968.log 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47292.log.gz 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42877.log 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38968.log.gz 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39895.log 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42877.log.gz 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66164.log 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39895.log.gz 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70438.log 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.66164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66164.log.gz 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78831.log 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70438.log.gz 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71483.log 2024-06-03T05:41:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78831.log.gz 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55801.log 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71483.log.gz 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77940.log 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55801.log.gz 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77091.log 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77940.log.gz 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39563.log 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77091.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77091.log.gz 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43004.log 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39563.log.gz 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39901.log 2024-06-03T05:41:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43004.log.gz 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67662.log 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39901.log.gz 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36429.log 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.67662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67662.log.gz 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40557.log 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36429.log.gz 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62918.log 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40557.log.gz 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71857.log 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62918.log.gz 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51516.log 2024-06-03T05:41:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71857.log.gz 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50207.log 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.51516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51516.log.gz 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46116.log 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50207.log.gz 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77599.log 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46116.log.gz 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48550.log 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77599.log.gz 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65180.log 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48550.log.gz 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43124.log 2024-06-03T05:41:53.385 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65180.log.gz 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50904.log 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43124.log.gz 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60359.log 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50904.log.gz 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74501.log 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.60359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60359.log.gz 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44407.log 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74501.log.gz 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63939.log 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44407.log.gz 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45709.log 2024-06-03T05:41:53.386 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63939.log.gz 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77151.log 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.45709.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45709.log.gz 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41486.log 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77151.log.gz 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68201.log 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41486.log.gz 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41315.log 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.68201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68201.log.gz 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65860.log 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41315.log.gz 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79254.log 2024-06-03T05:41:53.387 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65860.log.gz 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79254.log.gz 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40306.log 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74216.log 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40306.log.gz 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70292.log 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74216.log.gz 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62818.log 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70292.log.gz 2024-06-03T05:41:53.388 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78318.log 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62818.log.gz 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39896.log 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78318.log.gz 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54987.log 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39896.log.gz 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72606.log 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54987.log.gz 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63639.log 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72606.log.gz 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63639.log.gz 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40515.log 2024-06-03T05:41:53.389 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40366.log 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40515.log.gz 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63855.log 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40366.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40366.log.gz 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78150.log 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63855.log.gz 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79914.log 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78150.log.gz 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78108.log 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79914.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79914.log.gz 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41821.log 2024-06-03T05:41:53.390 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78108.log.gz 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69453.log 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41821.log.gz 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64101.log 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.69453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69453.log.gz 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78838.log 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64101.log.gz 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72089.log 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78838.log.gz 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74277.log 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72089.log.gz 2024-06-03T05:41:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log 2024-06-03T05:41:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74277.log.gz 2024-06-03T05:41:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39159.log 2024-06-03T05:41:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2024-06-03T05:41:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71223.log 2024-06-03T05:41:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39159.log.gz 2024-06-03T05:41:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54495.log 2024-06-03T05:41:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71223.log.gz 2024-06-03T05:41:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78532.log 2024-06-03T05:41:53.392 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54495.log.gz 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78192.log 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78532.log.gz 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40155.log 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78192.log.gz 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75773.log 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40155.log.gz 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40256.log 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.75773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75773.log.gz 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71344.log 2024-06-03T05:41:53.393 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40256.log.gz 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41978.log 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71344.log.gz 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75990.log 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41978.log.gz 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72009.log 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.75990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75990.log.gz 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64614.log 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.72009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72009.log.gz 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63007.log 2024-06-03T05:41:53.394 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64614.log.gz 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71387.log 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63007.log.gz 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76652.log 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71387.log.gz 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49156.log 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.76652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76652.log.gz 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77557.log 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49156.log.gz 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65622.log 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77557.log.gz 2024-06-03T05:41:53.395 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77257.log 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65622.log.gz 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78276.log 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77257.log.gz 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74908.log 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78276.log.gz 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73748.log 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74908.log.gz 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35044.log 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73748.log.gz 2024-06-03T05:41:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47353.log 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35044.log.gz 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71080.log 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47353.log.gz 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71858.log 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71080.log.gz 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38652.log 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.71858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71858.log.gz 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75351.log 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38652.log.gz 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40683.log 2024-06-03T05:41:53.397 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.75351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75351.log.gz 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73360.log 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40683.log.gz 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68917.log 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73360.log.gz 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51179.log 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.68917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68917.log.gz 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74042.log 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.51179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51179.log.gz 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46910.log 2024-06-03T05:41:53.398 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.74042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74042.log.gz 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40185.log 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46910.log.gz 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61360.log 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40185.log.gz 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39297.log 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64821.log 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.61360.log: /var/log/ceph/ceph-client.admin.39297.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39297.log.gz 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61360.log.gz 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62109.log 2024-06-03T05:41:53.399 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64821.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68719.log 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64821.log.gz 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39765.log 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.62109.log: /var/log/ceph/ceph-client.admin.68719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62109.log.gz 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68719.log.gz 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65323.log 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39765.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80166.log 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75011.log 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80166.log: /var/log/ceph/ceph-client.admin.65323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65323.log.gz 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80166.log.gz 2024-06-03T05:41:53.400 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69735.log 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.75011.log: /var/log/ceph/ceph-client.admin.45187.log 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75011.log.gz 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.69735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80502.log 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69735.log.gz 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.45187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78024.log 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45187.log.gz 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80502.log.gz 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77814.log 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59900.log 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77814.log: /var/log/ceph/ceph-client.admin.78024.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78024.log.gz 2024-06-03T05:41:53.401 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77814.log.gz 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49960.log 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70377.log 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.59900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59900.log.gz 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49960.log.gz 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64080.log 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75824.log 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64080.log: /var/log/ceph/ceph-client.admin.70377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70377.log.gz 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64080.log.gz 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80334.log 2024-06-03T05:41:53.402 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64841.log 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.75824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75824.log.gz 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.80334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80334.log.gz 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63638.log 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42137.log 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.63638.log: /var/log/ceph/ceph-client.admin.64841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64841.log.gz 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63638.log.gz 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64758.log 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42137.log: /var/log/ceph/ceph-client.admin.68071.log 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42137.log.gz 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77982.log 2024-06-03T05:41:53.403 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.64758.log: /var/log/ceph/ceph-client.admin.68071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64758.log.gz 0.0% 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68071.log.gz 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48780.log 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77982.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71947.log 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77982.log.gz 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73516.log 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48780.log: /var/log/ceph/ceph-client.admin.71947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71947.log.gz 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48780.log.gz 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51289.log 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.73516.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46454.log 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73516.log.gz 2024-06-03T05:41:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70948.log 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.51289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51289.log.gz 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46454.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77646.log 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46454.log.gz 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76890.log 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70948.log.gz 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.77646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77646.log.gz 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60320.log 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65790.log 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.76890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76890.log.gz 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.60320.log: 0.0% 92.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-06-03T05:41:53.405 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60320.log.gz 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79536.log 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.65790.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65790.log.gz 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67859.log 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.79536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47463.log 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79536.log.gz 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.67859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67859.log.gz 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40276.log 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47463.log.gz 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2024-06-03T05:41:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5/var/log/ceph/ceph-client.admin.40276.log: --verbose -- /var/log/ceph/ceph-client.admin.78539.log 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40276.log.gz 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39554.log 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.78539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.78539.log.gz 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70677.log 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39554.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39554.log.gz 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/ceph-client.admin.50843.log 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.70677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70677.log.gz 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58752.log 2024-06-03T05:41:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50843.log.gz 2024-06-03T05:41:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51139.log 2024-06-03T05:41:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.58752.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58752.log.gz 2024-06-03T05:41:53.408 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46372.log 2024-06-03T05:41:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.51139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51139.log.gz 2024-06-03T05:41:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67303.log 2024-06-03T05:41:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46372.log.gz 2024-06-03T05:41:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.67303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67303.log.gz 2024-06-03T05:41:53.408 INFO:teuthology.orchestra.run.smithi038.stderr: 90.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-06-03T05:41:53.408 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-06-03T05:41:53.409 INFO:teuthology.orchestra.run.smithi038.stderr: 90.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-06-03T05:41:53.409 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-06-03T05:41:53.422 INFO:teuthology.orchestra.run.smithi064.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-06-03T05:41:53.424 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-06-03T05:41:53.424 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m0.312s 2024-06-03T05:41:53.424 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m0.975s 2024-06-03T05:41:53.424 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.057s 2024-06-03T05:41:53.640 INFO:teuthology.orchestra.run.smithi038.stderr: 89.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-06-03T05:41:53.643 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-06-03T05:41:53.643 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m0.562s 2024-06-03T05:41:53.643 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m1.739s 2024-06-03T05:41:53.643 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.251s 2024-06-03T05:41:53.644 INFO:tasks.ceph:Archiving logs... 2024-06-03T05:41:53.644 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/remote/smithi038/log 2024-06-03T05:41:53.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-03T05:41:54.241 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/remote/smithi064/log 2024-06-03T05:41:54.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-03T05:41:54.431 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-06-03T05:41:54.432 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-03T05:41:54.443 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_dfcc4b3c2c55532c4c04fa47551c9df1dffdf746/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-06-03T05:41:54.444 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-03T05:41:54.444 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-03T05:41:54.447 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-03T05:41:54.529 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-03T05:41:54.539 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-03T05:41:54.539 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T05:41:54.542 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T05:41:54.556 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-06-03T05:41:54.559 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-06-03T05:41:54.561 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T05:41:54.561 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-06-03T05:41:54.561 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv01.front.sepia.ceph.com 3 6 377 14 +489us[ +427us] +/- 62ms 2024-06-03T05:41:54.562 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 -617us[ -679us] +/- 53ms 2024-06-03T05:41:54.562 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +306us[ +306us] +/- 68ms 2024-06-03T05:41:54.562 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:41:54.564 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T05:41:54.564 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-06-03T05:41:54.564 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv01.front.sepia.ceph.com 3 6 377 15 +376us[ +370us] +/- 62ms 2024-06-03T05:41:54.564 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 12 -743us[ -743us] +/- 53ms 2024-06-03T05:41:54.565 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +147us[ +147us] +/- 68ms 2024-06-03T05:41:54.565 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T05:41:54.565 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-03T05:41:54.574 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-03T05:41:54.575 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-03T05:41:54.583 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-03T05:41:54.705 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-03T05:41:54.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-03T05:41:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:41:54.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-03T05:41:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:41:54.856 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-03T05:41:54.868 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-03T05:41:54.933 INFO:teuthology.task.internal:Duration was 2033.310826 seconds 2024-06-03T05:41:54.934 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-03T05:41:54.974 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-03T05:41:54.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-03T05:41:54.977 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-03T05:41:55.017 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T05:41:55.019 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T05:41:55.453 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-03T05:41:55.454 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-06-03T05:41:55.454 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-03T05:41:55.477 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-06-03T05:41:55.477 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-03T05:41:55.522 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-03T05:41:55.522 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-03T05:41:55.525 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-03T05:41:55.590 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-03T05:41:55.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-03T05:41:55.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-03T05:41:55.764 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-03T05:41:55.775 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-03T05:41:55.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-03T05:41:55.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-03T05:41:55.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-03T05:41:55.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:41:55.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:41:55.865 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-06-03T05:41:55.891 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-06-03T05:41:55.917 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:41:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:41:55.942 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-03T05:41:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T05:41:55.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-03T05:41:55.983 INFO:teuthology.task.internal:Transferring archived files... 2024-06-03T05:41:55.983 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/remote/smithi038 2024-06-03T05:41:55.984 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-03T05:41:56.028 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-02_21:00:03-rados-squid-distro-default-smithi/7738436/remote/smithi064 2024-06-03T05:41:56.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-03T05:41:56.062 INFO:teuthology.task.internal:Removing archive directory... 2024-06-03T05:41:56.063 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-03T05:41:56.065 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-03T05:41:56.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-03T05:41:56.125 INFO:teuthology.task.internal:Not uploading archives. 2024-06-03T05:41:56.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-03T05:41:56.153 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-03T05:41:56.153 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-03T05:41:56.156 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-03T05:41:56.169 INFO:teuthology.orchestra.run.smithi038.stdout: 265304 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 3 05:41 /home/ubuntu/cephtest 2024-06-03T05:41:56.171 INFO:teuthology.orchestra.run.smithi064.stdout: 265316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 3 05:41 /home/ubuntu/cephtest 2024-06-03T05:41:56.172 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-03T05:41:56.182 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-03T05:41:56.223 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-06-03T05:41:56.225 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/valgrind} duration: 2033.310825586319 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-06-03T05:41:56.225 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-03T05:41:56.306 INFO:teuthology.run:FAIL